LILT's Green Says English AI Translators Fail Non-English Workflows
Ai4
Translating native prompts into English fails to improve non-English AI reasoning because models lack localized context and world knowledge in their training weights.
## Background
Published on YouTube by Ai4 on August 21, 2026, this talk features Spence Green, CEO of LILT. The video's description highlights how multilingual and contextual failure serves as a key limiter to global AI adoption, trust, and ROI in enterprise environments.
## TL;DR
Spence Green demonstrates that despite advances in machine translation, frontier AI models and agents continue to experience a significant performance gap in non-English languages. Empirical data reveals that over 75% of task failures stem from model-intrinsic reasoning and missing localized domain knowledge rather than language comprehension issues. Translating prompts back to English fails to resolve these deficiencies because it cannot retrieve missing training context. To build effective global AI systems, researchers and enterprise deployers must move away from translated benchmarks and establish native-language evaluation and reinforcement learning environments.
## Atmosphere & style
The presentation carries an authoritative, analytical, and professional tone tailored for an executive and technical audience. Spence Green delivers his insights with clear empirical backing, drawing directly from real-world enterprise case studies and red-teaming benchmarks. The narrative feels grounded and practical, shifting from identifying hidden operational risks to offering structured, strategic solutions for global AI deployment.
## Key takeaways
- [00:04](https://www.youtube.com/watch?v=62kGx4g52UI&t=4s) Many in the tech industry incorrectly assume that because machine translation is largely solved, AI agents are equally capable across all languages.
- [00:04](https://www.youtube.com/watch?v=62kGx4g52UI&t=4s) For high-stakes enterprise applications, such as medical communications at Boston Children's Hospital, domain-specific verification in non-English languages is critical to prevent dangerous errors.
- [01:52](https://www.youtube.com/watch?v=62kGx4g52UI&t=112s) Multi-turn conversational reasoning benchmarks show non-English task pass rates lagging behind English by 10% to 15%, with gaps reaching 25% to 30% on broader benchmarks.
- [01:52](https://www.youtube.com/watch?v=62kGx4g52UI&t=112s) Linguistic comprehension accounts for less than 1% of task failures in non-English contexts, while over 75% stem from intrinsic model factors like reasoning and domain knowledge.
- [01:52](https://www.youtube.com/watch?v=62kGx4g52UI&t=112s) Universal tokenizers require roughly double the tokens for non-English prompts compared to English, straining context length and worsening multi-turn performance degradation.
- [06:47](https://www.youtube.com/watch?v=62kGx4g52UI&t=407s) Non-English prompts are increasingly used in software engineering as non-developers build prototypes in their native languages.
- [06:47](https://www.youtube.com/watch?v=62kGx4g52UI&t=407s) On localized terminal-style coding benchmarks, English achieves an 85%–90% pass rate while non-English lags by roughly 30%.
- [09:24](https://www.youtube.com/watch?v=62kGx4g52UI&t=564s) Translating native-language prompts into English before feeding them to LLMs yields zero performance improvement and often degrades results slightly.
- [09:24](https://www.youtube.com/watch?v=62kGx4g52UI&t=564s) Translating surface text into English cannot recover missing localized cultural, regulatory, or legal context if it was never compressed into the model's weights during pre-training.
- [11:35](https://www.youtube.com/watch?v=62kGx4g52UI&t=695s) AI researchers should abandon translated benchmarks and focus on creating verified, native-language benchmarks across specialized fields to build targeted RL training environments.
- [11:35](https://www.youtube.com/watch?v=62kGx4g52UI&t=695s) Enterprise deployers must build native-language evaluation suites to detect operational failures before deploying AI systems across global markets.
## Detailed walkthrough
### [00:04](https://www.youtube.com/watch?v=62kGx4g52UI&t=4s) The Premise: Translation is Solved, but Agents Fall Short
Spence Green, co-founder and CEO of LILT, opens his talk by addressing a widespread misconception in the tech industry: because machine translation is largely viewed as a solved problem, many assume AI agents are equally capable across all languages. Green clarifies that this is simply not true. LILT works directly with frontier AI labs to perform red teaming on deployed agents, gathering empirical data to quantify performance disparities across languages and developing solutions to bridge those gaps. Beyond general AI models, LILT focuses heavily on content and agent verification for high-stakes enterprise use cases.
To illustrate the critical importance of domain-specific verification, Green shares an example from customer Boston Children's Hospital, where LILT verifies medical materials provided to non-English speaking patients. In Arabic, for instance, the phrase for "two years" and "since the age of two" can be identical in certain context structures; misinterpreting this nuance when recommending a chemotherapy regimen can lead to disastrous medical consequences. Similar high-stakes demands for expert-verified information exist across legal, regulatory, and corporate environments.
- **Speaker**: Spence Green (Co-founder & CEO of LILT)
- **Key Client Example**: Boston Children's Hospital (verifying medical forms and patient communications)
- **Core Thesis**: Machine translation progress does not automatically equalize multilingual AI agent capability.
---
[01:52](https://www.youtube.com/watch?v=62kGx4g52UI&t=112s) Quantifying the Performance Gap in Multi-Turn Reasoning
To demonstrate the capability gap between English and non-English tasks, Green shares empirical data from frontier benchmarks produced by LILT's red-teaming team. The frontier labs use these benchmarks to evaluate models and create reinforcement learning (RL) environments for further training. Looking at a multi-turn conversational reasoning benchmark (simulating complex scenarios such as instructing a travel agent to book flights), non-English languages consistently lag behind English. Across languages like Arabic, German, and Korean, the performance gap in task pass rates ranges from 10% to 15%, and up to 25%–30% across other broader benchmarks.
Crucially, deep task-failure analysis reveals that linguistic understanding is *not* the primary cause of these errors. Frontier models exhibit near-superhuman linguistic comprehension. In fact, less than 1% of task failures stem from language comprehension issues. Instead, over 75% of failures are caused by intrinsic model factors, such as underlying reasoning mechanics and world/domain knowledge.
- **Multilingual Gap**: 10% to 15% lower performance on conversational reasoning compared to English (and up to 25%–30% on broader tasks).
- **Failure Analysis**: $<1\%$ of task failures are linguistic; $>75\%$ stem from model-intrinsic knowledge, reasoning, and tokenization dynamics.
- **Root Causes**:
- **Token Density**: Universal, language-independent tokenizers require roughly double the tokens for the same prompt in non-English languages compared to English.
- **Context Length**: Languages like German (long compound words) or agglutinative languages like Finnish create higher token counts, which strains context windows.
- **Multi-Turn Degradation**: On complex tasks lasting 10+ turns, the performance gap widens significantly as token context grows.
---
[06:47](https://www.youtube.com/watch?v=62kGx4g52UI&t=407s) Coding Agents and Language Barriers: The Terminal Benchmark
Switching to software engineering, Green addresses the belief that coding is immune to multilingual issues because developers primarily work in English. In reality, non-English prompts are increasingly common as AI coding tools lower barriers to entry, enabling non-developer roles like product managers and designers to engage in "vibe coding" and build prototypes in their native languages. LILT developed a native terminal-style benchmark featuring prompts written originally in local languages to test agents on domain- and language-specific coding requirements—such as formatting Japanese text to fit fixed-width constraints on financial terminals.
Evaluating state-of-the-art frontier models (tracing progress from GPT-5.4 to GPT-5.5 over a 6–7 month period) shows clear overall capability gains due to targeted RL training. However, while the benchmark is practically "saturated" for English at an 85%–90% pass rate, non-English performance lags by roughly 30%. For lower-resource languages, performance drops off even more sharply.
- **Benchmark Focus**: Native-language prompts requiring localized output constraints (e.g., Japanese text length formatting on financial terminals).
- **Model Progress**: Significant pass-rate gains observed between GPT-5.4 and GPT-5.5 over a 6 to 9 month period.
- **English vs. Multilingual Gap**: Benchmark pass rates for English reach 85%–90%, maintaining a ~30% gap ahead of non-English languages.
---
[09:24](https://www.youtube.com/watch?v=62kGx4g52UI&t=564s) Why Translating Back to English Doesn't Fix the Problem
Green presents what he considers the most surprising finding of the presentation: testing the strategy of translating native prompts back into English before feeding them to the model. Given that translation is largely solved and frontier models do much of their internal reasoning in English or code, one might expect that translating input prompts into English would match native English performance. However, evaluations across leading models from OpenAI, Anthropic, and Google show that translating native-language prompts back to English yields no performance gain—and frequently causes slight performance degradation.
This occurs because LLMs compress broad world knowledge into a single neural network. If localized cultural, legal, social, or regulatory contexts are missing from the training data, translating the surface language into English cannot recover that missing domain knowledge. Thus, solving enterprise AI globally requires building native, localized context directly into systems rather than relying on translation layers.
- **Tested Workaround**: Translating native-language prompts into English prior to model reasoning.
- **Observed Result**: Zero performance improvement, with frequent slight degradation across OpenAI, Anthropic, and Google frontier models.
- **Core Limitation**: Surface-level English translation cannot retrieve localized context or regional knowledge that was never compressed into the model's weights during pre-training.
---
[11:35](https://www.youtube.com/watch?v=62kGx4g52UI&t=695s) Actionable Roadmap for Researchers and Enterprise Deployers
Green concludes by outlining practical steps for two key groups: researchers building foundational models and enterprise FTEs/engineers deploying them into operational workflows. Historically, AI development was centered around English because most researchers were native English speakers, and benchmark leaderboards (e.g., MMLU variations) relied on simple translations. To truly scale AI globally across governments, sovereign environments, and international enterprises, multilingual capability must become a first-class priority.
For researchers, the immediate focus must be creating high-quality, verified, native benchmarks across specialized fields like legal, finance, math, and science rather than relying on translated tests. Once performance gaps are accurately quantified through native benchmarks, targeted RL environments can be built to systematically lift model capabilities. For enterprise deployers, the priority is building localized evaluation suites in the target market's native language. Relying solely on English evals leaves teams blind to operational failures until adoption stalls in non-English regions.
- **Recommendations for AI Researchers**:
- Stop relying on translated benchmarks or basic translated MMLU suites.
- Build verified, native-language benchmarks in high-stakes domains (finance, legal, regulatory, science).
- Use native benchmarks to build RL environments that directly target reasoning gaps.
- **Recommendations for Enterprise Deployers**:
- Establish native-language evaluation pipelines rather than evaluating systems exclusively in English.
- Apply domain-specific post-training and fine-tuning against verified native evals to ensure operational readiness before global deployment.
## Notable quotes
> [00:00](https://www.youtube.com/watch?v=62kGx4g52UI&t=0s) If your AI only works in English, it doesn’t scale—it just demos well. — Spence Green
Your AI Isn’t Global. It Just Thinks It Is.
Why multilingual context—not bigger models—is a real bottleneck to enterprise AI ROI
As enterprises race to deploy AI globally, most assume that success in English equals readiness everywhere. It doesn’t.
2026 is the year to challenge a core blind spot in enterprise AI strategy: multilingual and contextual failure is now the primary limiter of global adoption, trust, and ROI—especially as agents and RAG systems move into real operational workflows. Models don’t fail because they lack fluency; they fail because they retrieve the wrong context, are validated against the wrong benchmarks, and are evaluated in ways that don’t reflect how businesses actually operate across markets.
Drawing from applied research and large-scale enterprise deployments, this session will show why AI systems that appear “production-ready” in Silicon Valley quietly break when deployed across languages, domains, and regions—and why leaders often don’t see the damage until adoption stalls or risks surface.
Develop a clear executive framework for assessing whether your AI is truly global-ready, how multilingual evaluation and context validation unlock adoption, and why the next wave of enterprise AI value will be driven not by bigger models, but by better evidence of where systems work—and don’t.
Outline
Intro - The Problem: The Uncomfortable Truth
Most enterprise AI deployments appear successful because they’re validated in English, by English speakers, on English data.
But that success collapses the moment AI touches:
international employees
non-English documents
local regulations
frontline operations outside Silicon Valley
“If your AI only works in English, it doesn’t scale—it just demos well.”
Spence Green, co-founder and CEO of LILT, argues that while machine translation is widely considered solved, multilingual AI agents suffer significant operational and reasoning deficits in non-English environments due to missing domain knowledge and tokenization mechanics rather than language comprehension errors.
The Multilingual Reasoning Deficit and Root Causes (1:52)
According to empirical red-teaming data presented by Green, frontier models exhibit a 10% to 15% deficit in multi-turn conversational reasoning pass rates across languages such as Arabic, German, and Korean compared to English, widening to 25% to 30% on broader benchmarks. These gaps create severe risks in high-stakes environments. For example, Boston Children's Hospital identified identical Arabic phrasing for 'two years' and 'since the age of two,' which could result in catastrophic dosing errors in chemotherapy regimens.
Crucially, Green reports that linguistic comprehension accounts for less than 1% of task failures, as frontier models demonstrate strong grammatical and lexical understanding. Instead, over 75% of failures stem from intrinsic reasoning and structural mechanics. Language-independent tokenizers require roughly double the tokens for non-English prompts, which strains context windows for compound-heavy or agglutinative languages like German and Finnish and significantly degrades performance over interactions lasting ten or more turns.
Disparities in Multilingual Coding Agents (6:47)
Although software development has historically centered on English, Green observes that AI coding tools have lowered technical barriers, allowing non-developers to engage in 'vibe coding' and prototype software using native-language prompts. To evaluate this dynamic, LILT developed a terminal benchmark requiring agents to handle localized output constraints, such as formatting Japanese text to fit fixed-width financial terminal displays.
While frontier models demonstrated overall progress from targeted reinforcement learning between GPT-5.4 and GPT-5.5 over a six-to-nine-month timeframe, benchmark pass rates for English plateaued near saturation at 85% to 90%. Non-English tasks lagged behind by approximately 30%, with lower-resource languages experiencing even steeper performance drops.
Limitations of English Translation Workarounds (9:24)
A common proposed remedy—translating non-English prompts into English before feeding them to the model—proves ineffective in practice. Green notes that evaluations across frontier models from OpenAI, Anthropic, and Google showed zero performance improvement when translating native inputs back to English, often resulting in slight performance degradation.
Green explains that large language models compress world knowledge directly into their neural weights. If specific regional, cultural, regulatory, or legal contexts were omitted from pre-training data, translating the surface text into English cannot recover that missing domain knowledge, demonstrating that translation layers cannot substitute for foundational multilingual training.
Roadmap for Native Evaluation and Deployment (11:35)
To address these disparities, Green advises AI researchers to abandon translated evaluation suites like translated MMLU tests in favor of native, expert-verified benchmarks in high-stakes fields such as finance, law, regulation, and science. Quantifying performance gaps through native benchmarks allows developers to build targeted reinforcement learning environments that address actual reasoning deficits.
For enterprise practitioners, Green emphasizes the necessity of establishing native-language evaluation pipelines in target operational markets. Because English-only evaluations leave teams blind to regional failures, enterprises must apply domain-specific fine-tuning and post-training against verified native test suites prior to international deployment.
Green concludes that global AI deployment cannot succeed through translation layers alone, emphasizing that closing the performance gap requires integrating native cultural and domain knowledge into model training and enterprise evaluation suites.
## Background
该视频由频道 Ai4 于 2026 年 8 月 21 日发布在 YouTube 平台，记录了 LILT 联合创始人兼 CEO Spence Green 的主题演讲。页面简介指出，企业往往误以为在英语环境中验证成功的 AI 已具备全球化落地能力，深入探讨了多语言情境缺失而非模型规模如何成为制约企业 AI 投资回报率（ROI）与实际应用的核心瓶颈。
## TL;DR
Spence Green 指出，尽管机器翻译已高度成熟，但前沿 AI Agent 在处理非英语多轮任务时仍普遍存在 10% 至 30% 的性能落后。分析表明，失败根源并非纯粹的语言学理解（占比不足 1%），而是语言无关分词器导致的 Token 膨胀、长上下文衰减以及本地法规和常识知识的缺失。实验证明‘翻译回英文再推理’的直觉方案不仅无效甚至会导致性能衰退，企业和研究人员必须基于本土原生语境构建评测基准与强化学习对齐，才能实现真正的全球化落地。
## Atmosphere & style
演讲整体展现出清醒、务实且极具洞察力的技术布道风格，直接打破了硅谷对大模型全球通用性的盲目乐观。Spence Green 借助扎实的基准数据、医疗高危场景的具象案例以及反直觉的实验发现层层剖析，语调沉稳而一针见血。整场分享节奏紧凑，让听众在直面多语言 Agent 落地硬伤的同时，获得了兼具底层技术机理与企业实战视角的清晰行动指南。
## Key takeaways
- [00:04](https://www.youtube.com/watch?v=62kGx4g52UI&t=4s) 机器翻译技术成熟并不等同于 AI Agent 具备多语言可靠性，在医疗与法规等高风险场景下，多语言跨语境操作必须经过专家严格验证方可落地。
- [02:06](https://www.youtube.com/watch?v=62kGx4g52UI&t=126s) 在前沿模型多轮推理基准中，非英语与英语存在 10% 到 30% 的性能鸿沟；失败归因中纯语言理解失误不足 1%，超 75% 源于模型内在知识储备与推理逻辑缺陷。
- [04:27](https://www.youtube.com/watch?v=62kGx4g52UI&t=267s) 语言无关分词器导致 Token 密度严重失衡，德语、阿拉伯语等语言的 Token 消耗远高于英语，加剧了模型的上下文处理负担。
- [04:27](https://www.youtube.com/watch?v=62kGx4g52UI&t=267s) 长周期 Agent 交互中存在复合错误放大效应，在 10 轮以上的多轮交互中，非英语任务的通过率会出现雪崩式下滑。
- [06:47](https://www.youtube.com/watch?v=62kGx4g52UI&t=407s) 在代码生成与终端基准（Terminal Bench）中，非英语与英语存在约 30% 的性能差距，模型频频在 i18n 编码、字符集与本土格式转换上失效。
- [09:24](https://www.youtube.com/watch?v=62kGx4g52UI&t=564s) “将非英语 Prompt 翻译回英文再推理”的直觉解法被证伪，实验表明该做法非但没有提升表现，反而会导致性能轻微下降或持平。
- [09:24](https://www.youtube.com/watch?v=62kGx4g52UI&t=564s) 多语言 Agent 的核心瓶颈在于知识压缩与情境丢失，若模型预训练中缺失特定地域的真实工作流与上下文，仅靠语言转换无法无中生有。
- [11:35](https://www.youtube.com/watch?v=62kGx4g52UI&t=695s) 模型研发实验室应摒弃简单的机翻 Benchmark，构建专家验证的本土原生基准（Native benchmarks），并基于原生语境环境施加强化学习（RL）对齐。
- [11:35](https://www.youtube.com/watch?v=62kGx4g52UI&t=695s) 企业部署者严禁将纯英文评估体系直接套用于多语言业务流，必须在垂直领域构建本土原生评测集并实施后训练（Post-training）以保障 ROI。
## Detailed walkthrough
### [00:04](https://www.youtube.com/watch?v=62kGx4g52UI&t=4s) 破除迷思：翻译已基本解决，不代表 AI Agent 在多语言环境中同样可靠
在企业级 AI 的全球部署中，存在一个普遍而危险的认知误区：既然机器翻译技术已高度成熟，那么自主 AI Agent 在各类语言环境下的行动能力也应当是一致的。LILT 联合创始人兼 CEO Spence Green 指出，事实绝非如此。LILT 的核心业务之一是为大型跨国企业在多语言环境中验证内容与 AI Agent，并与前沿大模型实验室（Frontier Labs）紧密合作开展红队测试（Red Teaming），通过发现与量化缺陷来协助优化模型能力。
在医疗、法律和监管等高风险场景下，内容验证的必要性显而易见。以波士顿儿童医院（Boston Children's Hospital）为例，LILT 负责为不懂英语的就诊患者验证关键医疗表单。在阿拉伯语中，“两岁”（two years）与“从两岁起”（since the age of two）使用的是完全相同的短语表达；如果 AI 处理该表单时未能准确消歧，并在推荐化疗方案等高危医疗流程中产生偏差，将会带来灾难性后果。在这些场景中，信息在具备可操作性（Actionable）之前，必须经过专家严格验证。随着 AI 从生成文本转向执行具体操作的 Agent 时代，验证多语言任务的可靠性变得更加紧迫。
```mermaid
flowchart TD
accTitle: Multilingual Agent Failure Mechanism
subgraph NaiveView["常见误区 (Naive View)"]
N1["翻译技术成熟"] --> N2["模型内部自动转换"] --> N3["Agent 在各语言能力相同 (错误结论)"]
end
subgraph RealityView["实际瓶颈 (Reality & Bottlenecks)"]
R1["原生多语言任务输入"]
R1 --> R2["Token 膨胀与密度差异"]
R1 --> R3["多轮对话上下文衰减"]
R1 --> R4["本地文化/法规/社会常识缺失"]
R2 & R3 & R4 --> R5["任务执行失败 (性能差距 10%~30%)"]
end
```
---
### [02:06](https://www.youtube.com/watch?v=62kGx4g52UI&t=126s) 多轮推理基准测试：性能差距的根源并非“语言理解”
为了量化 AI Agent 在多语言多轮交互中的真实表现，LILT 构建了前沿基准测试并提供给前沿实验室，用于模型评估及构建强化学习（RL）数据与环境。在多轮对话推理基准测试（模拟如呼叫旅行社根据复杂指令预订航班等真实场景）中，数据直观展示了英语与其他语言之间的巨大鸿沟。
以英语为基准，阿拉伯语、德语和韩语在完全相同的任务流程中，性能落后幅度均在 10% 到 15% 之间。Spence Green 强调，跨不同语种和多项基准测试来看，英语与非英语在前沿模型上的表现普遍存在 10% 到 25% 甚至 30% 的性能差距。然而，通过深入标注任务失败归因可以发现，核心瓶颈根本不在于模型的“语言理解能力”：
- 前沿模型在纯语言学理解上早已达到超人类水平。
- 在该基准测试中，因纯粹语言理解失误导致的任务失败占比**不足 1%**。
- 超过 **75% 的任务失败**源于模型本身的内在知识储备、推理机制以及处理多语言的底层逻辑缺陷（如潜在空间对齐不良 Latent space misalignment，以及根深蒂固的英语中心主义推理偏差）。
---
### [04:27](https://www.youtube.com/watch?v=62kGx4g52UI&t=267s) 分词密度与多轮上下文衰减：德语、阿拉伯语为何更易出错
深入模型底层机制，现代大模型普遍采用语言无关的分词器（Language-independent tokenizers），这直接导致了不同语言在“Token 密度”上的巨大差异。在相同的任务描述下，英语由于分词效率极高，所需的 Token 数量大约只有其他语言的一半左右；而德语因存在大量长复合词，阿拉伯语以及芬兰语等黏着语，其 Token 消耗量会成倍膨胀。
这种 Token 密度的失衡在复杂、长周期的 Agent 交互中被急剧放大：
- **短轮次与长轮次衰减**：在 6 到 10 轮的交互中，非英语任务已开始显现劣势；一旦交互进入 10 轮以上（10+ turns），任务通过率（Pass rate）出现雪崩式下滑，英语与非英语之间的差距稳定拉大到 10%~15%。
- **复合错误放大**：语言本身的物理表征特征（Token 膨胀）加剧了长上下文处理负担，结合模型内部对特定国家的法律制度、社会习俗、文化背景及行业监管知识的欠缺，构成了多语言 Agent 任务失败的主要推手。
---
### [06:47](https://www.youtube.com/watch?v=62kGx4g52UI&t=407s) 编程 Agent 的多语言困境：从 Terminal Bench 看 30% 的鸿沟
AI 编程助手（如代码生成 Agent）常被误认为只需要支持英语，因为传统观念认为软件工程师普遍使用英语工作。但 Spence Green 指出这一假设早已过时——AI 编程工具正在降低开发门槛，大量非英语母语的产品经理、设计师正借助“氛围编程”（Vibe coding）直接构建产品原型。
LILT 推出了面向多语言交互的终端基准测试 **Terminal Bench**（一种类似 Harbor 风格的基准，通过输入原生语言 Prompt 要求 Agent 在终端环境中构建项目并生成符合要求的输出）。例如，在日文 Prompt 要求将数据以日文字符按固定宽度排版至金融终端的测试中，模型频频在国际化编码（i18n）、字符集处理、多语言交互环境以及数据格式转换上出现硬伤。
| 评估维度 | 英语环境表现 | 非英语环境（日/韩/阿等）表现 |
|:--- |:--- |:--- |
| **基准饱和度** | 达 85% ~ 90%，基本属于“已解决”（Solved）状态 | 仍处于攻坚阶段，存在约 **30% 的性能差距** |
| **模型迭代演进** | 从 GPT-5.4 迭代至 GPT-5.5 性能持续攀升 | 虽然 6-7 个月内 RL 训练带来显著提升，但绝对差距仍巨大 |
| **低资源语种** | 表现优异 | 语料密度越低，性能下滑越剧烈 |
---
### [09:24](https://www.youtube.com/watch?v=62kGx4g52UI&t=564s) “翻译回英文再推理”的假设为何彻底破灭
面对多语言性能差距，行业内一种直觉性的解决方案是：“既然机器翻译已经解决，且大模型底层主要在英语和代码构成的潜在空间中进行推理，何不直接将非英语任务的 Prompt 自动翻译回英语，让模型在它最擅长的英语中推理并完成任务？”
LILT 针对 OpenAI、Anthropic 和 Google 的顶级前沿模型进行了该实验，结果得出了全场最引人深思的结论：
- **翻译回英文非但没有提升表现，反而导致了轻微的性能下降（Performance degradation）或基本持平。**
- 这一反直觉结果证明，多语言 Agent 瓶颈的本质并非语言翻译转换，而是**知识压缩与情境丢失**。大模型可以被视为将海量世界知识压缩进单一神经网络的系统；如果模型在预训练和对齐阶段根本没有吸收特定语言、特定地域的本地化真实工作流与上下文知识，单纯在输入端做语言翻译转换根本无法无中生有。
> “如果你的 AI 仅在英语中有效，它并没有真正具备扩展能力——它只是演示做得很漂亮。”（“If your AI only works in English, it doesn’t scale—it just demos well.”）
---
### [11:35](https://www.youtube.com/watch?v=62kGx4g52UI&t=695s) 面向研究员与企业部署者的落地框架与本土 Eval 策略
要让 AI 真正赋能全球企业、主权机构并渗透到实际生产力场景中，多语言能力必须从边缘功能转变为“一等公民”（First-class citizen）。Spence Green 为两类核心从业者提出了清晰的行动框架：
```mermaid
flowchart LR
accTitle: Action Framework for Global AI
subgraph Researchers["模型研究人员 (Frontier Labs / Researchers)"]
R_A["拒绝单一英文/浅层翻译 Benchmark"] --> R_B["构建真实本土原生基准 (Native Tasks)"]
R_B --> R_C["基于原生环境搭建 RL 训练与对齐"]
end
subgraph Deployers["企业部署工程师 (FTDEs / Practitioners)"]
D_A["拒绝仅依赖英文 Eval 评估全局"]
D_A --> D_B["构建目标语种的原生 Verified Evals"]
D_B --> D_C["在垂直受限领域针对性 Post-training / RL"]
end
```
- **给模型研究人员（Researchers）的建议**：
- 当前各大前沿模型发布时附带的 Model Card，其列出的 12 项顶级基准（金融、法律、科学、数学等）绝大多数是纯英语的，所谓多语言 MMLU 也多是简单的机器翻译版本。
- 核心突破点在于构建经过严格专家验证的**本土原生任务基准（Native benchmarks）**，量化真实性能落差，随后基于原生语境构建强化学习（RL）环境进行定向能力对齐。只要精准量化问题并施加 RL，模型能力便能在数月内实现跃升。
- **给企业部署工程师与全职开发者（FTDEs / Deployers）的建议**：
- 严禁将纯英文评估体系套用于跨国企业或以非英语（如法语）为主营环境的业务流程中。
- 必须在本地原生语境中构建高可信度、经过专家校验的 Eval 评测集；在垂直受限领域中，基于本土数据环境进行后训练（Post-training）和强化学习，才能确保 AI Agent 在全球真实业务流中具备真正的投资回报率（ROI）与可用性。
## Notable quotes
> [00:04](https://www.youtube.com/watch?v=62kGx4g52UI&t=4s) 翻译已基本解决，不代表 AI Agent 在多语言环境中同样可靠 — Spence Green
> [00:04](https://www.youtube.com/watch?v=62kGx4g52UI&t=4s) 在这些场景中，信息在具备可操作性（Actionable）之前，必须经过专家严格验证。 — Spence Green
> [02:06](https://www.youtube.com/watch?v=62kGx4g52UI&t=126s) 跨不同语种和多项基准测试来看，英语与非英语在前沿模型上的表现普遍存在 10% 到 25% 甚至 30% 的性能差距。 — Spence Green
> [02:06](https://www.youtube.com/watch?v=62kGx4g52UI&t=126s) 因纯粹语言理解失误导致的任务失败占比不足 1%。 — Spence Green
> [02:06](https://www.youtube.com/watch?v=62kGx4g52UI&t=126s) 超过 75% 的任务失败源于模型本身的内在知识储备、推理机制以及处理多语言的底层逻辑缺陷 — Spence Green
> [04:27](https://www.youtube.com/watch?v=62kGx4g52UI&t=267s) 在相同的任务描述下，英语由于分词效率极高，所需的 Token 数量大约只有其他语言的一半左右 — Spence Green
> [04:27](https://www.youtube.com/watch?v=62kGx4g52UI&t=267s) 一旦交互进入 10 轮以上（10+ turns），任务通过率（Pass rate）出现雪崩式下滑，英语与非英语之间的差距稳定拉大到 10%~15%。 — Spence Green
> [09:24](https://www.youtube.com/watch?v=62kGx4g52UI&t=564s) 翻译回英文非但没有提升表现，反而导致了轻微的性能下降（Performance degradation）或基本持平。 — Spence Green
> [09:24](https://www.youtube.com/watch?v=62kGx4g52UI&t=564s) 如果你的 AI 仅在英语中有效，它并没有真正具备扩展能力——它只是演示做得很漂亮。（If your AI only works in English, it doesn’t scale—it just demos well.） — Spence Green
> [09:24](https://www.youtube.com/watch?v=62kGx4g52UI&t=564s) 多语言 Agent 瓶颈的本质并非语言翻译转换，而是知识压缩与情境丢失。 — Spence Green
> [11:35](https://www.youtube.com/watch?v=62kGx4g52UI&t=695s) 核心突破点在于构建经过严格专家验证的本土原生任务基准（Native benchmarks），量化真实性能落差，随后基于原生语境构建强化学习（RL）环境进行定向能力对齐。 — Spence Green
> [11:35](https://www.youtube.com/watch?v=62kGx4g52UI&t=695s) 严禁将纯英文评估体系套用于跨国企业或以非英语（如法语）为主营环境的业务流程中。 — Spence Green
## Mentioned
Spence Green, LILT, Ai4, 波士顿儿童医院 (Boston Children's Hospital), OpenAI, Anthropic, Google, Terminal Bench, Harbor, MMLU, GPT-5.4, GPT-5.5
## Video description
Spence Green - CEO, LILT
Your AI Isn’t Global. It Just Thinks It Is.
Why multilingual context—not bigger models—is a real bottleneck to enterprise AI ROI
As enterprises race to deploy AI globally, most assume that success in English equals readiness everywhere. It doesn’t.
2026 is the year to challenge a core blind spot in enterprise AI strategy: multilingual and contextual failure is now the primary limiter of global adoption, trust, and ROI—especially as agents and RAG systems move into real operational workflows. Models don’t fail because they lack fluency; they fail because they retrieve the wrong context, are validated against the wrong benchmarks, and are evaluated in ways that don’t reflect how businesses actually operate across markets.
Drawing from applied research and large-scale enterprise deployments, this session will show why AI systems that appear “production-ready” in Silicon Valley quietly break when deployed across languages, domains, and regions—and why leaders often don’t see the damage until adoption stalls or risks surface.
Develop a clear executive framework for assessing whether your AI is truly global-ready, how multilingual evaluation and context validation unlock adoption, and why the next wave of enterprise AI value will be driven not by bigger models, but by better evidence of where systems work—and don’t.
Outline
Intro - The Problem: The Uncomfortable Truth
Most enterprise AI deployments appear successful because they’re validated in English, by English speakers, on English data.
But that success collapses the moment AI touches:
international employees
non-English documents
local regulations
frontline operations outside Silicon Valley
“If your AI only works in English, it doesn’t scale—it just demos well.”
[Source](https://www.youtube.com/watch?v=62kGx4g52UI)
Source quotes
[0:04] Spence Green: 翻译已基本解决，不代表 AI Agent 在多语言环境中同样可靠
[0:04] Spence Green: 在这些场景中，信息在具备可操作性（Actionable）之前，必须经过专家严格验证。
[2:06] Spence Green: 跨不同语种和多项基准测试来看，英语与非英语在前沿模型上的表现普遍存在 10% 到 25% 甚至 30% 的性能差距。
[2:06] Spence Green: 因纯粹语言理解失误导致的任务失败占比不足 1%。
[2:06] Spence Green: 超过 75% 的任务失败源于模型本身的内在知识储备、推理机制以及处理多语言的底层逻辑缺陷
[4:27] Spence Green: 在相同的任务描述下，英语由于分词效率极高，所需的 Token 数量大约只有其他语言的一半左右
[4:27] Spence Green: 一旦交互进入 10 轮以上（10+ turns），任务通过率（Pass rate）出现雪崩式下滑，英语与非英语之间的差距稳定拉大到 10%~15%。
[9:24] Spence Green: 翻译回英文非但没有提升表现，反而导致了轻微的性能下降（Performance degradation）或基本持平。
[9:24] Spence Green: 如果你的 AI 仅在英语中有效，它并没有真正具备扩展能力——它只是演示做得很漂亮。（If your AI only works in English, it doesn’t scale—it just demos well.）
[9:24] Spence Green: 多语言 Agent 瓶颈的本质并非语言翻译转换，而是知识压缩与情境丢失。
[11:35] Spence Green: 核心突破点在于构建经过严格专家验证的本土原生任务基准（Native benchmarks），量化真实性能落差，随后基于原生语境构建强化学习（RL）环境进行定向能力对齐。
[11:35] Spence Green: 严禁将纯英文评估体系套用于跨国企业或以非英语（如法语）为主营环境的业务流程中。
Source quotes
[0:00] Spence Green: If your AI only works in English, it doesn’t scale—it just demos well.
Original transcript
[0:04] Hey everybody, how you doing? Um, I'm Spence
Green. I'm uh one of the co-founders and the
[0:08] CEO of LIILT. We uh verify content for
large enterprises in other languages.
[0:15] So what I'm going to talk a little bit
[0:20] um because there's a perception that um because
translation is for the most part a solved problem
[0:26] that agents are equally capable in all languages
and this is not true. Um so one of the parts of
[0:31] our company we do red teaming against uh deployed
agents working mostly with the frontier labs to
[0:37] make the models more capable in other languages.
uh and I'm just going to show you some results
[0:42] to quantify uh this performance difference that I
claimed and then tell you a little bit about how
[0:48] to fix it. So I mentioned that our business
one part of our business does verification.
[0:54] We actually think about verification for both AI
generated content but we think about it also for
[1:00] agents. Um and so this verification problem in
very high stakes use cases is pretty obvious.
[1:06] So one of our customers is Boston Children's
Hospital. We verify uh medical content for
[1:12] patients when they come into the hospital if they
don't speak English. Um so here's an example of a
[1:16] form. This is fictitious but uh you get the point
which is uh in this particular form if you don't
[1:22] get the ambiguity right in Arabic um uh that you
know in Arabic the phrase for two years and since
[1:29] the age of two are the same phrase. So if you
don't get that correctly uh and you're doing a
[1:34] uh recommending a chemotherapy treatment this can
have disastrous consequences. So there are lots of
[1:39] examples for this in regulatory context, legal
context, medical context where you have to get
[1:44] information that's verified by an expert um before
it's actionable. So now if we talk about agents
[1:52] and agentic actions and agents taking action um
do they perform as well in uh other languages
[2:01] as they do in English and do we have any need for
verification? So that's the that's the premise of
[2:06] this talk which is um because translation for the
most part is a solved problem um do agents perform
[2:12] equally well in all languages for the types of
jobs that we want to use them for. So I'm going to
[2:18] show two uh frontier benchmarks that we produce.
So we've got a team that does red teaming against
[2:24] Frontier models. We produce benchmarks. Those
benchmarks are then used by the labs to evaluate
[2:30] their models and then to produce uh RL data and
RL environments to improve the models. So we're
[2:36] going to look at two benchmarks. On the left is an
example of a multi-turn conversational reasoning
[2:43] benchmark. So you know like you call up a travel
agent and you give them some instructions and they
[2:47] book a flight. And on the right is agentic coding.
So this is like using cloud code. So you give
[2:53] a coding agent uh some instructions to produce
something and it runs and gives you an output. So
[3:00] let's start with the multi-turn reasoning and it
turns out you can actually quantify this. So this
[3:06] is a benchmark we built for multi-turn reasoning.
Um the yellow bar is or excuse me it's sort of
[3:11] white whitish is English. Um and then you've got
three other languages Arabic, German, and Korean.
[3:20] And so what you see is the performance gap to
English for the very same benchmark is anywhere
[3:25] from 10 to 15%. And you can see this across a lot
of different benchmarks. You know, you could pick
[3:30] three other languages if you wanted. You could
pick 10 other languages, you could pick some other
[3:34] benchmark. For the most part, you'll see gaps
of anywhere between 10 to 25 or 30% in frontier
[3:42] performance on English versus other languages. And
the issue is not actually language understanding.
[3:48] these models are superhuman in terms of their
linguistic understanding. That's not the
[3:53] underlying issue and you can quantify what the
issue is. So um uh the if you if you go in and
[4:00] have people annotate the task failures um less
than 1% of the task failures in this particular
[4:06] benchmark have to do with language. For the
most part they have to do with the model's
[4:10] knowledge and issues related to how the model
reasons and how the model processes language.
[4:15] Um, and so that's the uh green bar here. So the
you know over 75% of the issues across languages
[4:21] have to do with the model itself. They don't have
anything to do with linguistic understanding. And
[4:27] so here are here are some of the issues that you
get into. Um tokenization related issues. Models
[4:31] these days use language independent tokenizers.
Um and so if you don't tokenize languages uh
[4:37] you know have enough uh awareness of different
orthographies and languages, you can get into
[4:41] tokenization issues which I'll just show you in
a minute. Um these models for the most part today
[4:46] they reason in something like English/CM computer
code. Um and so there's latent space misalignment
[4:52] and how the models reason. Um and then getting
into Englishcentric reasoning based on the model's
[4:57] knowledge of the task that it's trying to solve.
And you can actually see this. So on the left hand
[5:02] side is the token density across tasks for this
particular benchmark. And what you'll see is that
[5:07] English is a little bit unusual and that the
number of tokens per task is roughly half what
[5:12] it is in other languages. So you take the same
task um the the task description in English is
[5:19] roughly half the tokens of the very same task
in other languages. This is obviously going to
[5:23] uh flex you know if you're doing German you see
here like lots and lots of tokens German has very
[5:29] long words um Arabic somewhere in there. If you
get some language like Finnish or something that
[5:34] has a glutenative is very long words you'll get
even more tokens. The other thing that you see is
[5:39] because the uh tokens uh the number of tokens
in the language uh differs you get into tasks
[5:47] that have more turns and the longer the number
of turns and the longer the token context of
[5:53] the task the lower the accuracy. So the uh on the
right hand side the bars um the set of four bars
[6:00] to the left that's 6 to 10 turns and then if you
get to 10 plus turns you see again this really big
[6:07] gap emerge between the white bar which is English
and other languages which lag by about 10 to 15%
[6:13] in task pass rate. Um so so there um again the the
takeaway here is that the linguistic understanding
[6:23] is not the problem with the model. It's intrinsic
characteristics of both the language itself and
[6:28] how it exacerbates how the model operates.
But then the knowledge encoded in the model
[6:33] uh about the legal context, social context,
cultural context, regulatory context across
[6:39] different countries those attack uh account
for the majority of the task errors. Okay,
[6:47] so now let's switch to coding. And this one,
this one's puzzling. You would think that most
[6:50] software engineers speak English and therefore
um you know they interact primarily in English,
[6:57] but this is not true. People use these coding
assistants in all kinds of languages. In fact,
[7:02] this opens up um software engineering to people
who weren't software engineers before. So you
[7:08] have product managers co, you know, vibe coding
prototypes. You have designers building things.
[7:13] And these are all people that may not necessarily
speak English, but they're producing code. Um, and
[7:18] so this is the terminal bench benchmark. And it's
it is what you think it is. It's a Harbor style
[7:23] benchmark where an engineer would type a prompt
to build something and then the output would uh
[7:29] uh be produced. And we've built this benchmark so
that it exacer the prompts are all written native
[7:36] in that language and it has to produce outputs of
things that are um more specific to that language.
[7:42] So this is a particular prompt um where uh the
output is Korean and it has to fit within a fixed
[7:49] width on uh like a financial terminal. Excuse
me. Is it Japanese? Yes, it's Japanese. You're
[7:56] correct. [snorts] Um okay. And uh so you can see
the types of issues that show up here. Um there's
[8:04] internationalization related issues. Those get
into sort of encoding and um and characters. Um
[8:10] the instruction interaction environment, the
language that you're interacting with. Um uh
[8:15] you get uh text conversion, data conversion type
issues. These are all encoded in the benchmark.
[8:24] And this is more current. So this is we just we're
actually about to publish this benchmark. So, it's
[8:29] got um it doesn't have GPT 5.6 on here, but it's
got everything else right at the frontier. Um and
[8:35] you see that the models are getting better. So,
you see the progress from GPT 5.4 to 5.5. Um quite
[8:41] a bit of improvement over just uh you know, a six
or seven month time period. This, you know, goes
[8:46] to show you can pretty much RL anything and it'll
get better. Um but on this particular benchmark,
[8:54] um the English performance, this benchmark is
saturated. it's up in like 85 90%. So right now,
[8:59] even though there's been significant
improvement over the last 6 to9 months
[9:04] on these different languages, the gap to
English stands at about 30%. Right now,
[9:09] it's pretty much a solved benchmark for English.
It's not even close to solved um for these other
[9:14] languages. And this is just three languages
we're showing here. As soon as you start to
[9:18] get into lower density, lower resource languages,
the performance drops off pretty considerably.
[9:24] Now this I think is the most interesting chart
maybe in the whole talk. Um and let me say what
[9:30] it says here. I mentioned at the beginning
of the talk that translation for the most
[9:33] part is a solved problem. So why don't you take
these native tasks if the if the systems reason
[9:39] pretty much in English and they uh reason in
something like English or code just take the
[9:46] prompts and the tasks and translate them back
to English and run them and see what happens.
[9:52] So what you see here is across all the different
model types. So here we've got um the the frontier
[9:58] model from OpenAI Anthropic and Google if you take
the tasks the native tasks translate them back to
[10:05] English and run them um actually you see slight
performance degradation um or roughly comparable
[10:11] performance. So it doesn't it doesn't help at
all to move them back into English. In fact it
[10:16] can hurt a little bit which is not what you would
expect. You would expect that if your, you know,
[10:21] prior belief is that translation is pretty much
a solved problem. These models reason in English.
[10:26] Terrific. Just translate them back to English.
Let the model reason in the language that it
[10:30] knows best and it should solve the problem. And
that doesn't turn out to be the case. And for the
[10:35] most part, the reason is because of the in the
way that the information is encoded um and the
[10:40] way that it um you sort of think about it as these
models are compressing a bunch of knowledge into
[10:47] uh into a single um into a single neural network.
And if that knowledge of the particular context
[10:53] and region is just simply not present in the
model, you can't get the problem right. So
[10:57] there's a lot of work happening right now in our
company and across the labs to build more native
[11:03] representative tasks of the types of things that
people are doing in work across the world because
[11:09] um if the goal is to use AI to transform
work everywhere to wire it into governments,
[11:14] companies everywhere else then multilingual
becomes a first class problem whereas for most
[11:18] of the history of AI the work was done in English
because most of the researchers spoke English. Um,
[11:23] so it's a really exciting time to be working in
multilingual AI because the need to have these
[11:29] models perform capably across languages
is greater than it's ever been before.
[11:35] Okay, so what does this mean for people who are
doing AI work? And I roughly divide this into
[11:42] researchers who are building models and then
FTDEs who are deploying models. If you're a
[11:47] researcher building models, the real key here
is having representative native tasks. Most of
[11:54] the benchmarks that exist in the world right
now, like if you take any model release and
[11:58] look at the model card that comes out, there's
always a table and it shows like 12 benchmarks
[12:03] and then it compares it against, you know, four
other models, those are all in English today.
[12:08] um even the multilingual sometimes they'll be
like MMLU multilingual and that's really just a
[12:15] very simple translation task. So all of the hard
frontier benchmarks in finance, legal, you know,
[12:21] GDP, VAW, science and math, those are all in
English. So the first step and what a lot of work
[12:26] is going into right now is building native tasks
and native benchmarks so you can actually quantify
[12:32] what the performance gap is to English. Once you
can quantify it, then you can start to RL against
[12:36] it. So if you're a researcher, the key part is
building um building highquality benchmarks that
[12:42] are verified and that you can trust and then
building um RL environments against those to
[12:48] improve the capability of the language. If you're
an FTE and you're building chat bots or you're
[12:54] building agents or you're deploying these into
enterprise systems, then it re really comes down
[12:58] to an eval um uh uh an eval activity, which is if
the only eval that you're building are in English,
[13:06] but your target is a global company or a
company that operates primarily in French,
[13:12] then you're not really getting the whole story.
So again, it comes down to building verified evals
[13:19] and benchmarks in that native language um that
you know are true and then you can quantify uh
[13:27] performance gaps then you know post-raining rling
things um those techniques all work uh especially
[13:35] if you're working in a very limited domain but
you've got to be able to construct the environment
[13:39] natively uh as opposed to just working on English.
So that's point number three which is as as you
[13:44] saw on this slide uh here there's been a lot
of progress as soon as you sort of quantify the
[13:49] problem and then start doing research against it
you can make really good progress but the key is
[13:54] to quantify the problem so that you can formulate
um tasks that you can RL against and improve task
[14:00] performance. I think there's a huge amount of
work to do across languages. Like I said, if the
[14:04] goal of AI is to wire it into every country,
every company, every sovereign environment,
[14:10] then you've got to make these models work
equally well in every country in the world. And
[14:14] that's never really been a first class goal of AI
historically. Um, so it's a really exciting time
[14:20] to be working on this this problem. Okay, that's
my talk. Thanks a lot. Finished a bit early.
