Indeed Automates Translation Pipelines Across 35 Locales

Why AI Isn't the Hardest Part of Language Operations With Indeed — Ai4

Ai4

The company routes content through automated quality scoring engines to bypass human reviewers while capping brand risk.

AI-assisted summary of Why AI Isn't the Hardest Part of Language Operations With Indeed — original source. Check the original for full context.

Source published:

Detailed notes

## Background This presentation was delivered by Ari Karampour, AI Strategist at Indeed, and published on YouTube by the Ai4 channel on August 16, 2026. According to the page description, the session explores the architecture behind an AI-powered language operating model, demonstrating how automated intake, AI translation, quality estimation, and decision engines combine with human oversight to build trusted, scalable, and autonomous language operations.

## TL;DR To handle tens of millions of translation characters across 35 locales without creating human bottlenecks or risking brand trust, Indeed shifted its focus from simple AI translation generation to an orchestrated, trusted delivery model. The architecture employs a three-layer system that automates content intake and contextual routing, performs AI translation enriched with style and terminology assets, and evaluates outputs via automated Quality Estimation and decision engines. Content identified as high-confidence and low-risk is delivered instantly, while high-risk or low-confidence content is routed to human experts. Edits and performance data from every step are fed back into the system to refine decision thresholds and retrain models, transforming human linguists into strategic system architects.

## Atmosphere & style The presentation is crisp, pragmatic, and highly authoritative, delivering an architectural deep dive into enterprise localization technology. Ari Karampour speaks with a clear, strategic focus, framing complex operational bottlenecks as elegant system design challenges. Watching it feels like a practical blueprint walkthrough for technical leaders, balancing realistic operational risks with a forward-looking vision for autonomous workflows.

## Key takeaways - [00:04](https://www.youtube.com/watch?v=CUEq9f92uVo&t=4s) Language operations should be designed as a configurable, modular orchestration architecture where LLMs, SaaS tools, and technology layers can be swapped without breaking workflows. - [00:48](https://www.youtube.com/watch?v=CUEq9f92uVo&t=48s) The primary operational hurdle in scaling translation is no longer content generation, but bridging the trust gap between over-reviewing (creating human bottlenecks) and under-reviewing (risking brand integrity). - [00:48](https://www.youtube.com/watch?v=CUEq9f92uVo&t=48s) Human oversight cannot scale alongside continuous AI output, shifting the core challenge from translation generation to trusted automated delivery. - [05:55](https://www.youtube.com/watch?v=CUEq9f92uVo&t=355s) Trustworthy automated delivery depends on systematically embedding workflow controls for pre-translation context, terminology compliance, and risk escalation. - [05:55](https://www.youtube.com/watch?v=CUEq9f92uVo&t=355s) To gain organizational adoption, the AI platform must prove risk detection reliability to language teams while guaranteeing faster turnaround times to business requesters. - [08:11](https://www.youtube.com/watch?v=CUEq9f92uVo&t=491s) The end-to-end architecture uses an automated Quality Estimation (QE) engine to compute confidence scores for translated content prior to delivery. - [08:11](https://www.youtube.com/watch?v=CUEq9f92uVo&t=491s) A systematic decision engine automatically delivers high-confidence, low-risk content while routing low-confidence or high-risk content to human review tracks. - [08:11](https://www.youtube.com/watch?v=CUEq9f92uVo&t=491s) Operational data across delivery, locales, and human edits continuously feeds back into the pipeline to recalibrate decision rules and fine-tune models. - [11:13](https://www.youtube.com/watch?v=CUEq9f92uVo&t=673s) Layer One (Auto-Intake & Contextual Analysis) evaluates risk levels, business priority, and locale needs to route high-stakes content down different paths than routine text. - [11:13](https://www.youtube.com/watch?v=CUEq9f92uVo&t=673s) Layer Two (AI Translation & Refinement) dynamically injects terminology databases, translation memories, and past reviewer feedback into the LLM context window. - [11:13](https://www.youtube.com/watch?v=CUEq9f92uVo&t=673s) Layer Three (Quality Estimation & Risk Routing) runs automated checks, anomaly detection, and background human sampling to validate outputs without delaying instant delivery. - [15:11](https://www.youtube.com/watch?v=CUEq9f92uVo&t=911s) Automation shifts human linguists from manual routing and repetitive corrections to high-leverage roles like governance policy definition, exception handling, and model training. - [15:11](https://www.youtube.com/watch?v=CUEq9f92uVo&t=911s) Human corrections and feedback are captured directly from exception handling to retrain LLMs, refine prompt design, and adjust scoring thresholds.

## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=CUEq9f92uVo&t=4s) Introduction and the Vision for Flexible AI Operations

Ari Karampour, AI Strategist at Indeed, introduces himself and his role leading the technology and automation team within language services. His responsibilities span strategy implementation, technology stack management, and data analytics. Setting the stage for his 20-minute presentation, Karampour frames language operations not merely as a translation task, but as an orchestration story centered around technology integration and workflow automation.

The foundational vision was to construct a fully configurable AI framework built with flexible workflows. Karampour wanted a system where individual components—whether specific Large Language Models (LLMs), SaaS software providers, or underlying tech stack modules—could be swapped out seamlessly without breaking the pipeline. Balancing this modularity required achieving "responsible speed," delivering content with high confidence, and strategically directing human effort to where it adds maximum value within the automation loop.

- **Role:** AI Strategist leading Technology and Automation for Language Services at Indeed. - **Core Objective:** Modular AI language architecture with interchangeable LLMs, SaaS tools, and tech stack layers. - **Key Principles:** Responsible speed, confident delivery, and optimal utilization of human expertise.

---

[00:48](https://www.youtube.com/watch?v=CUEq9f92uVo&t=48s) The Core Challenge: Scale, Capacity, and the Trust Gap

The fundamental challenge Indeed faced was scaling automated translation without sacrificing trust. As generative AI models unlocked the ability to translate content continuously and instantly across dozens of locales, the sheer volume of translated text quickly outpaced the review capacity of any human team. Indeed handles tens of millions of translation characters annually across 35 locales, covering diverse content types ranging from sensitive legal documentation to support articles and UI labels.

This massive output created a strategic paradox: if human review is applied to every output, humans become the primary bottleneck, destroying the scale advantage of AI. Conversely, if content is pushed through with too little review, governance and brand trust collapse. Realizing that manual oversight could not scale alongside output, Indeed shifted its focus from a *translation generation problem* to a *trusted delivery and orchestration problem*, making the AI layer responsible for deciding when to automate and when to route to a human.

``` +-------------------------------------------------+ | Tens of Millions of Characters | | 35 Locales & Tiers | +-------------------------------------------------+ | v +---------------------------------------------+ | AI Translation Engine (High Output) | +---------------------------------------------+ | +--------------+--------------+ | | v v [ Too Much Review ] [ Too Little Review ] • Human Bottleneck • Governance Risk • Destroys Scale • Loss of Brand Trust | | +--------------+--------------+ | v +---------------------------------+ | THE TRUST GAP | | Requires Automated Orchestration| +---------------------------------+ ```

- **Scale Drivers:** Tens of millions of translation characters per year across 35 locales. - **Content Complexity:** Variable risk levels across legal documents, marketing material, support articles, and UI elements. - **The Bottleneck:** Human reviewer capacity is inherently capped, making 100% manual review impossible. - **The Trust Gap:** The dilemma between slowing down AI with excessive manual checks versus taking on brand and governance risks with zero-shot automation.

---

[05:55](https://www.youtube.com/watch?v=CUEq9f92uVo&t=355s) Defining Architecture and Addressing Dual Stakeholders

To establish trust across the organization, Indeed recognized that trustworthy delivery depends on every operational decision made within a workflow rather than translation accuracy alone. Systematic controls had to be embedded directly into the workflow architecture to address pre-translation context, model selection, terminology compliance, and sensitive content escalation. Karampour emphasizes that internal teams need full visibility into the end-to-end processing of every piece of content.

Building this operational model required earning the confidence of two distinct internal audiences. For internal language services and quality teams, the system had to prove it could reliably detect risks, escalate appropriate edge cases to human experts, and eliminate redundant checking. For business requesters across Indeed, the platform needed to guarantee faster turnaround times using AI while preserving the exact same quality benchmarks and accountability as traditional human workflows.

- **Workflow Governance Questions:** - Is the correct content entering the automated intake pipeline? - Does the translation model receive complete context and style rules? - Are terminology guidelines and policy constraints properly applied? - Is sensitive or legal content safely escalated? - Can internal teams audit and track every step in real time? - **Internal Language Teams:** Require proof that risk detection is reliable and human time is reserved for non-routine tasks. - **Business Requesters:** Demand accelerated turnaround without sacrificing content quality or accountability.

---

[08:11](https://www.youtube.com/watch?v=CUEq9f92uVo&t=491s) The End-to-End Language Operating Architecture

To operationalize trusted delivery, Indeed deployed a closed-loop, data-driven architecture. The workflow begins with an automated intake process that ingests requests across channels and performs orchestration analysis to determine optimal routing. Content moves into an AI translation layer—powered primarily by LLMs—and undergoes refinement via agentic routines, automatic post-editing, or contextual adjustments. Next, an automated Quality Estimation (QE) engine computes a confidence score for the translation.

This confidence score feeds into a systematic decision node that balances quality scores against business risk rules. High-confidence, low-risk content is delivered directly to the requester in a fully automated, hands-off transaction. High-risk or low-confidence content is diverted into specialized human review tracks. Crucially, data gathered at every stage—delivery metrics, locale performance, content type trends, and human review edits—feeds back into the decision pipeline to continuously recalibrate system rules and fine-tune models.

``` +-----------+ +------------+ +-----------+ +-------------+ | Auto |--->| Analyze & |--->| AI Model |--->| Refinement | | Intake | | Route | | (LLMs) | | & Context | +-----------+ +------------+ +-----------+ +-------------+ | v +-----------+ +------------+ +-----------+ +-------------+ | Instant |<---| High Conf. |<---| Decision |<---| Quality Est.| | Delivery | | / Low Risk | | Engine | | Scoring | +-----------+ +------------+ +-----------+ +-------------+ | | | v | +--------------+ | | Human Review | | +--------------+ | | +--------+--------+ | v +-----------------------+ | Continuous Feedback | | Loop & Data Analytics | +-----------------------+ ```

- **Automated Intake & Orchestration:** Direct ingestion of requester content followed by metadata and contextual analysis. - **AI Translation & Refinement:** LLM processing augmented by agentic workflows and automated post-editing (APE). - **Quality Estimation & Scoring:** Algorithmic evaluation of output confidence prior to delivery. - **Decision Engine Rules:** Instant delivery for high-confidence/low-risk content; targeted escalation for high-risk assets. - **Feedback Loop:** Performance data across all 35 locales is recycled back into the system to refine decision thresholds and retrain models.

---

[11:13](https://www.youtube.com/watch?v=CUEq9f92uVo&t=673s) Detailed Breakdown of the Three Architectural Layers

Karampour details the three functional layers that form the foundation of Indeed's language operating system. Layer One handles auto-intake and contextual analysis before translation begins, evaluating variables such as content channel, target audience, business importance, market-specific locale requirements, and failure risk levels. This ensures high-stakes assets like legal contracts follow fundamentally different workflow paths than routine support articles or user interface strings.

Layer Two focuses on AI translation, refinement, and contextual routing. To maximize model performance, the system dynamically feeds relevant terminology databases, translation memories, style guides, metadata, and past human feedback into the LLM context window. Layer Three oversees quality estimation, risk routing, and continuous validation. This final layer performs mandatory formatting and term checks, flags anomalies, calculates quality scores, and runs background sampling—evaluating translated samples concurrently without holding up real-time delivery.

- **Layer 1: Auto-Intake & Contextual Analysis** - Assesses content channel, business critical risk, target audience, and locale-specific market expectations. - Applies strict pathing rules (e.g., legal vs. support content). - **Layer 2: AI Translation, Refinement & Routing** - Supplies contextual assets: terminology, translation memory, style guides, and past reviewer feedback. - Determines whether to reuse existing translations, post-edit, or trigger agentic refinement. - **Layer 3: Quality Estimation & Risk Routing** - Executes automated checks for formatting, completeness, and terminology adherence. - Performs anomaly detection and runs quality estimation models. - Conducts background human-in-the-loop sampling to gather validation data without blocking instant delivery pathways.

---

[15:11](https://www.youtube.com/watch?v=CUEq9f92uVo&t=911s) Transforming the Human Role for Autonomous Operations

As automation takes over routine translation and orchestration, the role of human linguists and localization specialists at Indeed undergoes a fundamental transformation. Rather than performing manual file transfers, chasing business requesters, or manually correcting repetitive translation errors, human experts step into high-leverage strategic roles. Their focus shifts toward defining system policies, managing edge cases, and training models.

Human expertise is directly integrated into the architecture through exception handling and continuous feedback loops. When exceptions occur, human edits are captured to retrain LLMs and refine decision rules. Additionally, feedback mechanisms actively monitor for model hallucinations, ensuring that high-risk content receives rigorous controls while routine content moves unimpeded. Karampour concludes by emphasizing that while AI provides scaling power, it is a well-designed architecture—backed by targeted human review—that creates the trust required for autonomous language operations.

- **Legacy Human Tasks (Eliminated):** Manual routing, tracking requests across channels, and manually fixing repetitive errors. - **Evolved Human Roles:** Setting global governance policies, evaluating complex exceptions, and providing feedback data. - **LLM Maintenance:** Human edits are systematically recycled to fine-tune prompts, adjust scoring thresholds, and retrain LLMs. - **Core Summary:** "AI is powerful... but trust is what lets language operations use it at scale."

## Notable quotes > [15:11](https://www.youtube.com/watch?v=CUEq9f92uVo&t=911s) AI is powerful... but trust is what lets language operations use it at scale. — Ari Karampour

## Mentioned Ari Karampour, Indeed, Ai4, YouTube

## Video description Ari Karampour - AI Strategist | Localization Technology & Automation, Indeed

AI can translate content at a scale no team could ever manually review, but how do you know when it is safe to automate? As organizations adopt AI across their operations, the challenge is no longer generating content. It is building the technology stack, orchestration layers, and workflow automation that enable trusted, end-to-end delivery. This session explores the architecture behind an AI-powered language operating model, showing how automated intake, AI translation, quality estimation, decision engines, and human oversight work together to determine when content can be delivered automatically and when human review is still required. Attendees will learn practical design principles for building AI workflows that combine confidence signals with business rules to scale automation while maintaining quality, governance, and trust. Join this session to learn how trusted AI architectures can move organizations beyond simple automation toward truly autonomous language operations.

[Source](https://www.youtube.com/watch?v=CUEq9f92uVo)

中文简报

Indeed 实现 35 个语言区域翻译流程自动化 该公司通过自动化质量评分引擎分流内容，在控制品牌风险的同时省去人工审校环节。 为了在覆盖 35 个语言区域、处理数千万字符翻译的同时避免产生人工瓶颈或危及品牌信任，Indeed 将重心从单纯的 AI 翻译生成转向了经过编排的可信交付模式。该架构采用了一个三层系统：实现内容接入与上下文路由的自动化，结合风格与术语资产执行 AI 翻译，并通过自动化质量评估与决策引擎对输出结果进行评估。被判定为高置信度且低风险的内容会即时交付，而高风险或低置信度的内容则分流给人工专家。各环节的编辑与性能数据均会反馈给系统以优化决策阈值并重训模型，从而使人工语言专家转型为战略层面的系统架构师。 Indeed 的 AI 战略师 Ari Karampour 提出，语言运营不仅是翻译任务，更应构建为高度可配置且模块化的编排架构，确保 LLM、SaaS 工具与底层技术模块能够无缝替换且不中断工作流。 (0:04) 面对覆盖 35 个地区语言、年均数千万字符的翻译体量，核心挑战已从内容生成转变为弥合“信任差距”——既要避免全面人工审核沦为效率瓶颈，也要防止过度放任自动化引发品牌与合规风险。 (0:48) 端到端闭环架构通过自动化质量预估（QE）引擎为翻译内容计算置信度分数，并依托决策引擎实现高置信度、低风险内容全自动即时交付，同时将边缘风险案例路由至人工专家通道。 (8:11) 语言专家的职能由日常机械纠错转向制定治理策略与管理异常用例，人工修改数据持续反馈至管道中，用于动态重新校准决策规则并微调模型。 (15:11)

Reading summary

According to Indeed AI Strategist Ari Karampour, the central challenge in modern language operations is not raw AI translation generation, but orchestrating a modular architecture that bridges the trust gap between automated scale and human capacity. The Scalability Paradox and the Trust Gap (0:48) Indeed processes tens of millions of translation characters annually across 35 locales, spanning assets from low-risk UI strings and support articles to high-risk legal documents. While generative AI models can produce continuous, near-instant translations, the volume generated quickly overwhelms manual review teams. Karampour notes that applying 100% human oversight negates the speed and scalability advantages of AI, turning human reviewers into an operational bottleneck. Conversely, bypassing human review entirely risks catastrophic breakdowns in brand trust, accuracy, and legal compliance. This dynamic creates an operational "trust gap." To resolve it, Indeed shifted its strategy from viewing localization as a generation problem to treating it as an orchestration and routing problem, tasking the system with determining when automation is sufficient and when human intervention is mandatory. Addressing Dual Stakeholders Through Workflow Governance (5:55) Establishing organizational trust required Indeed to construct systematic controls embedded directly into the workflow architecture rather than relying on after-the-fact translation checks. The framework audits context ingestion, model selection, terminology compliance, and policy guidelines while maintaining real-time auditability across all assets. Karampour emphasizes that the platform must satisfy two distinct internal groups. Internal language services and quality teams require proof that automated risk detection reliably flags edge cases, eliminating redundant checks and reserving human time for critical tasks. Concurrently, business requesters demand accelerated turnaround times without sacrificing the accountability and quality benchmarks established by traditional human localization workflows. The Three-Layer Closed-Loop Operating Architecture (8:11) To operationalize automated translation safely, Indeed structured its language platform into three interconnected layers. Layer One handles auto-intake and contextual analysis, evaluating incoming requests by channel, business importance, target audience, and risk level to route content onto appropriate workflow tracks. Layer Two manages AI translation and refinement, dynamically supplying Large Language Models with context, glossaries, translation memory, and style guides while deploying agentic routines and automated post-editing. Layer Three oversees automated quality estimation (QE), anomaly detection, and risk routing. An algorithmic QE engine scores translation confidence against established business risk thresholds: high-confidence, low-risk content is delivered instantly to requesters, while low-confidence or high-risk assets divert to human review tracks. Crucially, background human-in-the-loop sampling validates output without interrupting instant delivery paths, feeding performance metrics and edit data back into the system to fine-tune models and recalibrate rules. Reframing Human Specialists as Strategic Operators (15:11) As automated orchestration takes over routine translations and file routing, the operational function of linguists and localization specialists fundamentally shifts. Instead of performing manual intake, chasing requesters, or repeatedly correcting minor translation errors, human experts occupy strategic oversight roles. Within this architecture, specialists define global governance policies, manage complex edge cases, and evaluate system anomalies. Every human edit and exception resolution is captured in closed feedback loops to retrain LLMs, refine prompt engineering, and update decision-scoring thresholds, ensuring models are systematically hardened against hallucinations and recurring errors. Karampour concludes that while AI models provide the power to scale translation exponentially, long-term operational autonomy depends entirely on a robust, modular architecture that directs human expertise strictly toward risk mitigation and continuous model governance.

精炼阅读摘要

Indeed人工智能战略师Ari Karampour指出，语言运营的核心瓶颈并非模型生成能力本身，而在于如何通过自动化编排与风险分流弥合“信任鸿沟”；唯有构建以质量预估与数据闭环为核心的模块化架构，才能在保障品牌安全的同时释放大语言模型的规模化交付价值。 规模扩张与信任鸿沟的权衡 (0:48) 在Indeed每年跨35个区域语言市场处理数千万翻译字符的业务背景下，生成式人工智能技术实现了连续且即时的内容翻译，但译文总量的激增迅速超越了人工团队的审核承载上限。Karampour分析认为，这种海量输出引发了运作上的两难困境：如果对所有AI译文进行全量人工审核，人力产能将成为新的系统瓶颈，彻底抵消AI带来的规模与速度优势；而若完全免除人工审核直接发布，则会导致合规监管风险失控，甚至损害品牌声誉。 面对敏感法律合同、客户支持文章与用户界面文案等差异巨大的业务风险级别，Indeed意识到单纯提高翻译吞吐量无法解决运营瓶颈。语言团队必须将工作焦点从“文本生成”转向“可信交付与流程编排”，利用智能决策系统决定内容是在全自动流程中直接交付，还是转由专业人工介入处理。 面向双重内部利益方的模块化治理 (5:55) 为了在组织内部建立对自动化运营的信任，系统必须同时满足两类诉求截然不同的利益相关方。对于内部语言服务与质量团队而言，系统必须证明其风险识别机制精准可靠，能准确升级异常边界案例，并消除繁琐的日常重复性审核；而对于发起翻译需求的业务部门，平台既要借助AI显著缩短周转周期，又必须维持与传统全人工流程完全相同的质量基准和问责标准。 Karampour强调，可信交付取决于工作流中的全链条运营决策而非单纯的模型翻译准确率。Indeed因而确立了高配置性与模块化原则，确保大语言模型（LLM）、SaaS软件提供商及底层技术栈组件均可随时无缝替换，避免技术绑定；同时在翻译前注入语境、术语规范及敏感内容分流规则，为内部团队提供对内容生命周期的实时审计追踪能力。 三层递进的自动化评估与决策架构 (11:13) Indeed落地了一套包含三层职能的闭环式语言操作系统。第一层为自动摄入与语境分析，系统在翻译前评估内容渠道、目标受众、商业关键度、特定区域市场要求及潜在故障风险，使高风险的法律资产与常规支持文档从源头上走入不同的处理路径。第二层聚焦AI翻译与语境路由，通过向LLM动态注入术语库、翻译记忆库、风格指南和历史反馈资产，辅以智能体（Agentic）流程与自动译后编辑（APE）以增强输出一致性。 第三层负责质量预估（Quality Estimation, QE）与风险路由。系统自动执行格式与术语合规检验，检测文本异常，并由QE算法实时生成置信度评分。决策引擎依据质量评分与业务风险规则分流：低风险高置信度内容直接全自动交付；高风险或低置信度内容则自动转入专门的人工专家审核队列。此外，系统通过后台异步抽样持续验证质量，避免因审核阻塞即时交付通路。 人工专家向高价值战略监督者的角色重塑 (15:11) 随着底层自动化承担了海量常规翻译与调度任务，Indeed的语言学家与本地化专家的职责发生了本质转变。人工团队不再消耗时间在手动传输文件、跨渠道跟进需求或机械修改重复错误上，而是转型为系统规则制定者、高风险异常处理者和模型维护者。 人工审查产生的修改数据与各语言市场的交付指标被直接回收至底层数据闭环中。这些反馈数据不仅用于微调提示词和重新训练LLM，还被用来校准决策引擎的置信度阈值并实时监控模型幻觉。Karampour总结认为，AI虽然赋予了业务极高的扩展速度，但唯有通过精密架构设计与针对性的人工介入闭环，才能建立起支撑自主化语言运营所必需的组织信任。 Karampour总结主张，语言运营的未来在于实现“负责任的速度”，即以灵活、可插拔的模块化系统结合质量预估模型，将宝贵的人力精准配置于高风险异常治理与数据反馈优化中，以此解决自动化扩张与品牌合规之间的根本张力。

Detailed notes — zh

## Background 本内容由频道 Ai4 于 2026 年 8 月 16 日发布在 YouTube 平台上，演讲者为 Indeed 语言服务技术与自动化 AI 策略专家 Ari Karampour。视频简介指出，企业在利用 AI 规模化翻译内容时，最大的挑战已不再是内容生成，而是构建集成了自动摄入、质量预估与决策引擎的技术架构与编排系统，以实现安全且可信的端到端交付。 ## TL;DR Indeed 语言服务团队负责人 Ari Karampour 详细拆解了如何构建一套高度可配置、以数据为驱动且兼顾人机协同的 AI 语言运营体系。他指出，语言本地化的瓶颈已从单纯的“翻译生成”转向“系统编排与信任机制”，企业必须通过三层架构精准平衡自动化直出与人工审核。系统通过摄入环境研判、上下文丰富化注入、自动化质量预估（QE）与并发抽样风控，实现了高置信度低风险内容全自动交付，并将低置信度高风险内容精准流转给人工专家。这一架构彻底解放了人类语言专家，使其角色从机械的译后编辑转变为定义治理规则、处置系统边缘异常及利用反馈数据反哺模型迭代。 ## Atmosphere & style 整场演讲呈现出严谨、务实且极具工程实操深度的专业技术大会氛围。Ari Karampour 的表达条理清晰、逻辑缜密，没有空洞的 AI 营销话术，而是以一线技术架构师的视角层层剖析系统落地中的现实痛点与破解方案。从揭示人工审核面对海量数据的无力感，到逐步铺开三层编排体系与闭环数据流，给听众带来一种拨开迷雾、化繁为简的系统设计美感与理性说服力。 ## Key takeaways - [00:04](https://www.youtube.com/watch?v=CUEq9f92uVo&t=4s) Indeed 构建语言运营体系的核心愿景是打造“完全可配置的 AI 架构”，在无需重构系统的前提下，可自由插拔和替换底层 LLM、SaaS 供应商及各类组件。 - [00:04](https://www.youtube.com/watch?v=CUEq9f92uVo&t=4s) 架构设计追求“负责任的速度”，在追求自动化交付效率的同时，以“人力价值的最优利用”为支点，将语言专家精准部署在最高价值的控制节点上。 - [02:09](https://www.youtube.com/watch?v=CUEq9f92uVo&t=129s) 生成式 AI 的产出规模已彻底打破人工逐行审查的上限，语言运营的核心挑战已从单纯的“翻译生成问题”转变为底层的“系统编排与决策问题”。 - [02:09](https://www.youtube.com/watch?v=CUEq9f92uVo&t=129s) 规模化落地的最大阻碍并非技术或模型本身，而在于如何建立系统信任——即在无法逐字检查的前提下，确保内容交付的安全与可靠。 - [04:52](https://www.youtube.com/watch?v=CUEq9f92uVo&t=292s) 自动化与人工审核之间存在微妙平衡：检查过多会导致系统重回人力瓶颈，审查过少则面临合规失控，解决方案是用完备的控制流代替“审查一切”。 - [04:52](https://www.youtube.com/watch?v=CUEq9f92uVo&t=292s) 运营架构必须同时建立双重信任：既要让语言质量团队确信系统能可靠拦截高危内容，也要让业务部门确信 AI 交付内容的质量标准与责任机制与人工时期一致。 - [08:11](https://www.youtube.com/watch?v=CUEq9f92uVo&t=491s) 自动化的每一次扩张都必须有数据背书，通过构建闭环数据回流通道，将交付与审核数据反哺决策引擎，持续优化路由规则与决策模型。 - [08:11](https://www.youtube.com/watch?v=CUEq9f92uVo&t=491s) Indeed 的自动化核心哲学明确界定：将可预测的确定性工作彻底自动化，将不确定性风险精准升级，并在涉及高敏或重大业务后果时保留人工判断。 - [11:13](https://www.youtube.com/watch?v=CUEq9f92uVo&t=673s) 架构第一层（摄入与分析）在调用 AI 前完成环境研判，依据内容类型、目标受众与错误严重程度，对法律、营销、UI 等不同资产进行定制化路由。 - [11:13](https://www.youtube.com/watch?v=CUEq9f92uVo&t=673s) 架构第二层（翻译与润色）贯彻“上下文决定模型表现”的原则，向 LLM 注入术语库、TM 及“历史人工语言学反馈”，智能决定直出或 Agentic 润色。 - [11:13](https://www.youtube.com/watch?v=CUEq9f92uVo&t=673s) 架构第三层（评估与路由）引入质量预估（QE）与幻觉检测，并在后台执行非阻塞的并发质量抽样（Sampling），在保障主干流水线畅通的同时监控大盘质量。 - [15:11](https://www.youtube.com/watch?v=CUEq9f92uVo&t=911s) 人类专家的角色从低价值的手工分流和机械改错，彻底转向系统治理策略定义、质量阈值维护，以及深入分析未达标的边缘异常案例（Exceptions）。 - [15:11](https://www.youtube.com/watch?v=CUEq9f92uVo&t=911s) 专家在审查系统异常时沉淀的高质量纠错数据，将直接回流并用于模型微调（Fine-tuning）与架构进化，形成人机协同正向循环。 - [15:11](https://www.youtube.com/watch?v=CUEq9f92uVo&t=911s) “可信自动化”的真谛在于系统不仅清晰知晓何时放行、何时拦截，更能“用数据充分解释二者的差异”，从而在风险可控的前提下释放规模化生产力。 ## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=CUEq9f92uVo&t=4s) 引言与愿景：构建灵活、负责且以人为本的 AI 语言运营架构 在 Ai4 大会上，Indeed 语言服务技术与自动化团队负责人 Ari Karampour 带来了一场关于技术集成、系统编排与工作流自动化的演讲，深入阐述了 Indeed 如何构建一套值得信赖的 AI 驱动型语言运营模式（AI-driven language operations model）。Ari 及其团队不仅负责制定整体战略，还主导了相关技术栈落地、技术运维以及底层数据分析支持。 Ari 指出，团队在启动该项目时的核心愿景是打造一个“完全可配置的 AI 架构”（fully configurable AI）。这个架构必须具备极高的工作流灵活性，允许团队在无需重构整体系统的前提下，轻松替换底层的大语言模型（LLM）、SaaS 供应商甚至技术栈中的各个独立组件。与此同时，架构设计必须平衡“负责任的速度”（responsible speed）与交付信心——既要追求自动化带来的敏捷交付，又要确保 AI 的翻译表现切实达到业务严苛的要求。而维系这一平衡的核心支点，在于“对人力价值的最优利用”（best use of human effort），即如何将语言专家放置在系统中最关键的节点上，实现人机协同的最大效能。 ### [02:09](https://www.youtube.com/watch?v=CUEq9f92uVo&t=129s) 规模与信任的悖论：从“翻译生成”转向“系统编排” 生成式 AI 的崛起为企业语言本地化带来了前所未有的扩张机遇。AI 产出语言内容的能力和规模，已经彻底超越了任何人工团队逐行审查的承载极限。持续翻译、即时交付、覆盖更多语种与市场，以及消除冗长的人工交接链路，在技术上都变得触手可及。然而，这随之引发了一个致命的现实困境：当翻译输出的速度与体量彻底吞没人工审核能力时，企业该如何确保交付安全？ 在 Indeed，语言团队每年需要处理数千万字符的翻译需求，覆盖全球 35 个语言区域（locales）。每一个本地化市场都有独特的文化语境与规范要求；翻译资产横跨多种内容类型，各类型内部还划分了细致的风险等级与优先级梯队；再叠加产研部门持续不断的产品更新与内容迭代，有限的人工审核资源迅速成为了交付瓶颈。 ```mermaid flowchart TD accTitle: Translation Problem vs Orchestration Problem subgraph Traditional["传统视角：翻译生成问题"] A["内容输入"] --> B["人工/机器翻译"] B --> C["全量人工逐字审核 (瓶颈)"] C --> D["交付上线"] end subgraph AIOrchestration["AI 运营视角：系统编排与决策问题"] E["自动化摄入 (35+ 区域/数千万字符)"] --> F{"AI 决策与路由层"} F -->|"高置信度 + 低风险"| G["全自动直出交付"] F -->|"低置信度 / 高风险"| H["精准升级人工专家审核"] end ``` 如果无法逐项检查每一句译文，系统该如何判断哪些内容可以直接放行，哪些必须立即拦截？Ari 强调，正是这一挑战让团队意识到，语言运营的本质已经不再是一个单纯的“翻译问题”，而是一个底层的“系统编排问题”（orchestration problem）。AI 在该体系中不再仅仅是翻译工具，更是核心的“决策层”（decision layer）。归根结底，“最艰难的部分并非引入 AI，而是如何建立信任”（"The hardest part for us wasn't AI, it was trust."）。 ### [04:52](https://www.youtube.com/watch?v=CUEq9f92uVo&t=292s) 弥合信任鸿沟：面向内部与业务端建立双重信任机制 在传统的人工审核与激进的全面自动化之间，存在着一条微妙的信任界限。若设置过多的人工检查，自动化系统就会被拖慢，人类重回效率瓶颈，AI 带来的规模化扩张红利荡然无存；反之，若审查不足，企业将直面合规治理失效与用户信任崩塌的巨大风险。因此，解决方案绝非“审查一切”，而是在 AI 工作流中构筑更健全的控制机制。 对于 Indeed 而言，可信交付取决于工作流中的每一个决策质量，而非仅靠译文本身的字面效果。在着手搭建技术架构前，团队必须在系统层面解答一系列关键问题： - 自动化入口摄入的内容是否准确无误？ - 系统是否为模型注入了完整的语境与上下文？ - 在入口处是否匹配了最合适的大模型或翻译服务？ - 专有名词术语表与企业策略控制是否被严格执行？ - 潜在的高危敏感内容是否能被可靠地升级拦截？ - 内部团队能否端到端全链路可视化监控系统每一步的运行状态？ Ari 总结道，构建这一运营模型必须同时赢得两种受众的“双重信任”： 1. **语言服务与质量内部团队**：确信系统具备严密的风险识别与拦截能力，只将真正需要人工介入的内容精准上报，并在低风险环节安全放行，减轻无意义的人工复核负担。 2. **企业内部业务需求方（全公司各业务线）**：确信通过 AI 加速生产的内容，在质量标准与责任追溯机制上与过去纯人工把关时期毫无二致。 ### [08:11](https://www.youtube.com/watch?v=CUEq9f92uVo&t=491s) 数据驱动的端到端语言运营架构与闭环反馈 为了将信任转化为系统能力，Indeed 搭建了一套以数据证据为核心支撑的自动化架构——自动化的每一次扩张，都必须有扎实的数据作为背书。 ```mermaid flowchart LR accTitle: End-to-End Language Operations Architecture subgraph Layer1["第1层：分析与路由"] A["自动摄入 Intake"] --> B["编排分析 Analysis"] B --> C["决策路由 Routing"] end subgraph Layer2["第2层：翻译与润色"] C --> D["LLM 翻译 Translation"] D --> E["智能润色与后审 Refinement"] end subgraph Layer3["第3层：评估与分流"] E --> F["质量与置信度打分 Scoring"] F --> G{"决策判定 Decide"} G -->|"高置信/低风险"| H["即时直出交付 Delivery"] G -->|"低置信/高风险"| I["人工专家审核 Review"] end H -.->|"闭环数据回流 Feedback Loop"| G I -.->|"审核数据回流 Feedback Loop"| G ``` 整套工作流始于全自动的摄入模块，无论需求方提交何种类型的内容，系统都会迅速启动编排分析并进行智能分流路由。随后进入 AI 翻译与润色阶段，利用 LLM、智能体（Agentic）工作流或上下文感知自动译后编辑进行精细化处理。完成翻译后，系统会对内容进行质量估算与置信度打分，并根据得分触发最终决策：达到直出标准的译文将无需人工干预直接交付给需求方；未达标或触发风控的译文则精准流转至专项人工审核通道。 Ari 特别指出了架构图中从“交付/审核”连回“决策引擎”的闭环虚线。团队通过埋点持续采集全链路数据：哪些语言区域表现最佳？哪些内容类型极难实现自动化交付？哪些场景可以高频直接放行？这些详实的数据反馈源源不断地回流至底层引擎，成为系统规则重构与决策模型迭代的基石。其核心哲学十分明确：“将可预测的确定性工作彻底自动化，将不确定性风险精准升级，并在涉及高敏或重大业务后果的节点保留适度的人工判断”。 ### [11:13](https://www.youtube.com/watch?v=CUEq9f92uVo&t=673s) 三层架构深度解析：从输入分析、上下文润色到风险路由 Ari 进一步拆解了该技术体系的三大核心分层： | 架构层级 | 核心模块 | 关键职责与技术机制 | |:--- |:--- |:--- | | **第一层：自动摄入与分析**<br>(Auto Intake & Analysis) | 摄入、分析与初始路由 | 在调用 AI 前完成环境研判：识别内容渠道、业务权重、目标受众、目标语言区域规范、潜在错误后果及质量等级要求（如法律合规文档、客服知识库文章、营销广告文案与 UI 界面标签遵循完全不同的工作流路由）。 | | **第二层：AI 翻译与智能润色**<br>(Translation & Refinement) | 语境注入与模型生成 | 秉承“模型效果取决于上下文丰富度”的理念，向 LLM 注入专用术语库、翻译记忆库（TM）、风格指南、元数据、本地化特定指令，以及**历史人工语言学反馈（Prior Feedback）**；系统在此阶段决策资产复用、直接翻译或智能体后编辑。 | | **第三层：质量评估与风险路由**<br>(Quality Estimation & Risk Routing) | 自动化 QE、风控与决策 | 结合业务规则与质量得分决定内容去向：执行术语合规与排版一致性校验、自动化质量预估（QE）、异常检测与模型对比、幻觉检测，以及**并发质量抽样与人在回路机制（Sampling & HITL）**。 | 在第三层中，并发抽样机制尤为关键：系统在不阻塞主干自动化交付流水线的前提下，会在后台并发抽取样本进行基准质量评估，将持续产生的数据用于监控大盘与验证模型健康度。规则引擎根据“高置信度 + 低风险 = 自动直出；高风险 = 人工接管”的逻辑，确保只有安全的内容才能进入全自动交付通道。 ### [15:11](https://www.youtube.com/watch?v=CUEq9f92uVo&t=911s) 重塑人类专家的角色：从手动分发转向策略定义与异常治理 伴随架构的演进，语言团队中人类专家的角色定位发生了深刻转变。传统的运营模式充斥着大量低价值劳动：人工分配工单、追着业务方确认上下文、在各个系统间手动搬运文件、日复一日地机械纠正重复性语法错误。 在高度自动化的新架构下，人类专家的精力被彻底解放并重新聚焦于高价值领域： - **制定与维护系统治理策略**：参与定义质量评估阈值、业务规则与风险控制红线。 - **专攻系统异常案例审查**：深入分析那些未通过自动化验证的边缘异常内容（Exceptions），找出系统判断失误的根本原因。 - **数据回流驱动模型进化**：将人工审查异常时产生的优质纠错数据，反哺用于架构微调（Fine-tuning）与大语言模型的持续重训。 Ari 总结道，在 Indeed 的语言服务实践中，“可信自动化”（Trusted Automation）的真谛是一个既懂得何时放手自动化、何时不该自动化，更能“用数据清晰解释二者差异”的智能系统。它能够以极快的速度处理常规内容，为关键核心资产套上严密的防护控制，并利用置信度打分精准捕获 LLM 幻觉。AI 固然强大，但唯有通过精妙的人机协同架构建立起坚实的信任体系，企业才能真正安全地释放规模化语言运营的全部潜力。 ## Notable quotes > [00:04](https://www.youtube.com/watch?v=CUEq9f92uVo&t=4s) 完全可配置的 AI 架构 — Ari Karampour > [00:04](https://www.youtube.com/watch?v=CUEq9f92uVo&t=4s) 负责任的速度 — Ari Karampour > [00:04](https://www.youtube.com/watch?v=CUEq9f92uVo&t=4s) 对人力价值的最优利用 — Ari Karampour > [02:09](https://www.youtube.com/watch?v=CUEq9f92uVo&t=129s) 系统编排问题 — Ari Karampour > [02:09](https://www.youtube.com/watch?v=CUEq9f92uVo&t=129s) The hardest part for us wasn't AI, it was trust. — Ari Karampour > [08:11](https://www.youtube.com/watch?v=CUEq9f92uVo&t=491s) 将可预测的确定性工作彻底自动化，将不确定性风险精准升级，并在涉及高敏或重大业务后果的节点保留适度的人工判断 — Ari Karampour > [11:13](https://www.youtube.com/watch?v=CUEq9f92uVo&t=673s) 模型效果取决于上下文丰富度 — Ari Karampour > [11:13](https://www.youtube.com/watch?v=CUEq9f92uVo&t=673s) 高置信度 + 低风险 = 自动直出；高风险 = 人工接管 — Ari Karampour > [15:11](https://www.youtube.com/watch?v=CUEq9f92uVo&t=911s) 可信自动化 — Ari Karampour ## Mentioned Ari Karampour, Indeed, Ai4, LLM, SaaS, TM, HITL ## Video description Ari Karampour - AI Strategist | Localization Technology & Automation, Indeed AI can translate content at a scale no team could ever manually review, but how do you know when it is safe to automate? As organizations adopt AI across their operations, the challenge is no longer generating content. It is building the technology stack, orchestration layers, and workflow automation that enable trusted, end-to-end delivery. This session explores the architecture behind an AI-powered language operating model, showing how automated intake, AI translation, quality estimation, decision engines, and human oversight work together to determine when content can be delivered automatically and when human review is still required. Attendees will learn practical design principles for building AI workflows that combine confidence signals with business rules to scale automation while maintaining quality, governance, and trust. Join this session to learn how trusted AI architectures can move organizations beyond simple automation toward truly autonomous language operations. [Source](https://www.youtube.com/watch?v=CUEq9f92uVo)

Source quotes

[0:04] Ari Karampour: 完全可配置的 AI 架构 [0:04] Ari Karampour: 负责任的速度 [0:04] Ari Karampour: 对人力价值的最优利用 [2:09] Ari Karampour: 系统编排问题 [2:09] Ari Karampour: The hardest part for us wasn't AI, it was trust. [8:11] Ari Karampour: 将可预测的确定性工作彻底自动化，将不确定性风险精准升级，并在涉及高敏或重大业务后果的节点保留适度的人工判断 [11:13] Ari Karampour: 模型效果取决于上下文丰富度 [11:13] Ari Karampour: 高置信度 + 低风险 = 自动直出；高风险 = 人工接管 [15:11] Ari Karampour: 可信自动化

Source quotes

[15:11] Ari Karampour: AI is powerful... but trust is what lets language operations use it at scale.

Original transcript

[0:04] Thank you everyone. Hi, I'm Ari and  I am an AI strategist and speaker. [0:12] I lead the technology and automation  team at Indeed for language services. [0:19] And my team and I also run implementation and  support of those strategies as well as the [0:25] tech stack and the data analytics that support  it which we'll talk about a little bit today. [0:34] Welcome and thank you for being here. So  I'm going to take you through a story. [0:40] It's a 20inut story about technology integrations  orchestration and workflow automation. [0:48] that powers a trusted AIdriven language  operations model that we have built at indeed. [0:58] So the story starts at the beginning always  with an idea and a concept. And for me, this [1:04] this was the overarching goal essentially, which  was I wanted to build something that was a fully [1:12] configurable AI that had flexible workflows  where we could swap out LLMs, where we could [1:20] swap out even SAS providers, where we could swap  out even parts of the tech stack fairly simply. [1:29] And then on the other end was how do we do this  with responsible speed. So speed was important [1:36] but it also needed to be responsible when  it comes to language services and how do [1:41] we deliver confidently meaning what we deliver  um we have the confidence that the AI has done [1:50] its job to our satisfaction and in the middle of  all this is the best use of human effort. How do [2:00] we use the people that we have within language  services to help us within this framework? [2:09] So there was a challenge  for us and the challenge is [2:14] how do you automate at scale without sacrificing  trust. So for us the AI became the decision layer [2:24] and that decision layer determined when we  added a human in the layer in the process [2:33] and then it also determined when we  could let automation do its thing. [2:38] So the hardest part for us  wasn't AI, it was trust. [2:45] Now AI provided us a great opportunity to scale. [2:51] So AI could produce more language content  than really any team could review and inspect. [2:59] So there was continuous translation available,  faster delivery. We could take on more languages [3:05] and local and there would be a lot less manual  handoff in the process. However, what we needed [3:14] to ask is what happens with when the translation  output exceeds our ability to review the content. [3:25] So this was no longer a problem of content  generation. It was a problem of trusted delivery. [3:35] And so more languages, more content is what  we're being faced with at Indeed. However, [3:43] within the AI portion, there were more models  and more decisions to be made by those models. [3:50] So at Indeed, we're looking at tens of  millions of translation characters per year. [3:56] 35 local and each of these local have specific  expectations and instructions within them. [4:05] As many content types as you can imagine  and risk levels within these content types [4:12] and tiers within those risk levels and then of  course continuous product and content updates. [4:20] And we only have so many human reviewers. So that  limits our capacity for review. So the question [4:30] becomes if we can't check everything then how do  we know what content moves and what content stops. [4:39] So this is when we realize that the  language operations has become more [4:45] of an orchestration problem and  not so much a translation problem. [4:52] There was also a trust gap, right? Manual  review meant that we could not scale. [5:00] So there was a fine line right between  too much review where the automations [5:08] that we have built and put in place essentially  slows down. The humans become the bottleneck [5:15] and the AI that we use to be able to scale  is no longer going to have a scale advantage. [5:23] Now on the other end, if we had too little  review of the content that was going through AI, [5:29] then we ran the risk of governance  and trust becoming an issue. [5:35] And so it didn't matter how fast AI  could go if those were the main concerns. [5:42] So the answer wasn't that we wanted to  review everything and every piece of content. [5:47] The answer became, how do we put better  controls into our workflows with AI? [5:55] So trust is more than translation quality for us. [5:59] A trustworthy delivery depends on every decision  in the workflow, not the translation itself. [6:06] And so trust here is going to be  produced by an architecture that [6:11] could support this model. And that is the  architecture that I'm going to show you. But [6:17] before building the architecture systematically we  had to answer a lot of questions in order to frame [6:23] this architecture. Some of them are here but these  are just very limited. So we wanted to see is the [6:29] right content even entering the workflow because  it was automated. There were no human checks. [6:37] Did the system receive the right context? Was  the right model or service selected at the [6:44] at the initial uh entry point? Were terminology  and policy controls properly applied? [6:53] Was sensitive content escalated  properly? Which is very important indeed. [7:02] And more importantly internally, can our team  see what's happening every step of the way? [7:12] And I started to realize there are two  audiences here which meant that there were [7:17] two kinds of trust. So building this operating  model means that we have to earn the trust [7:26] of the internal teams in language services, the  quality teams of course but also the business. So [7:34] internally we need to create confidence that  the system can catch the risks and escalate [7:41] the right work to people and reduce checking  where we don't need people to actually check. [7:49] But for our business requesters, which  is essentially the rest of Indeed, [7:55] we needed to provide them with the confidence that  content can be produced faster with AI, but that [8:02] we would still ensure the same level of quality  and accountability as when humans tracked it. [8:11] And so this is the architecture put in  place. And it's really in our architecture [8:18] where automation can only expand when we  have the evidence to support it expanding. [8:26] Okay? And that evidence is datadriven. So we begin  with the intake model that's completely automated [8:35] which makes it faster and easier for all our  requesters regardless of the type of content [8:40] that they are requesting. And that goes through  an analysis. That analysis is orchestration [8:49] and from there there's a routing and that's  the decision layer that I was speaking of [8:53] right systematically it's making decisions for  us and then next is where we go to the AI part [9:02] translation which is mainly done through  LLMs and then refining the translation. So [9:09] that could be an agentic approach. It could be  automatic posts editing. It could be contextual [9:17] and then scoring a confidence scoring. So how well  did the AI do in the first part of the translation [9:25] in the refinement of the translation? And now it's  going to get a score. And that score then moves [9:31] into a decision point. So the decision point is  are we going to go ahead and deliver the content [9:37] instantly or are we going to need a review cycle  for that and what that review cycle looks like. [9:45] And so if it's delivery it automatically goes to  the requesttor and it's completely done handoff. [9:53] But if it's review then it goes  to specific avenues for review. [9:58] Now I have a dotted line back to  decide and I want to tell you why. [10:03] Through the data that we're gathering  on every step of the way, we then use [10:10] content that is going to be delivered or reviewed  back into the decision pipeline of the workflow. [10:18] So for me it was important to understand how  much of the content was delivered automatically, [10:23] how much was not, which loces did best,  which loces did worse, what type of content [10:29] can be delivered automatically, what type of  content rarely gets delivered automatically. [10:36] And all of that data goes back  into rebuilding the system. [10:42] So we wanted to automate what was predictable. We  knew that we were translating a day in and day out [10:48] all the time and we knew with fairly good amount  of confidence we could we could automate that. [10:54] And then we wanted to escalate what was uncertain  and we wanted to have the right amount of human [11:01] judgment in the process for anything that  was really sensitive or consequential. [11:06] And at the foundation was data. Right? So  this is very datadriven this architecture. [11:13] So there's three layers here. Layer one, I  want to go into that architecture a little bit [11:18] more with you. Layer one is the auto intake and  analysis. So before the transition even begins, [11:24] before the AI really takes over, the system  needs to know what type of decision it is making, [11:32] right? So content type and channel uh the context  the audience and the business importance within [11:41] Indeed, the local and the market expectations  and what risks were there if the output was [11:48] wrong for instance and then what was the required  quality level. So you can imagine a legal document [11:57] versus a support article versus a marketing  headline versus a UI label do not necessarily [12:04] go down the same path and workflow. So that  layer one is the intake analyze and routing. [12:15] Layer two which is the AI becomes the  translation refinement and routing. So [12:22] the model is only as useful as the context  within its path around it. Right? So [12:28] we needed to make sure context was all there. That  meant terminology, memory, style guides, metadata, [12:35] local instructions, and most importantly, prior  feedback. What I mean by prior prior feedback [12:41] is anything that has already been reviewed  that we know through linguistic efforts and [12:49] through the people that are doing this all the  time that they have given us that also goes into [12:55] the model and in refining the model and building  the model. And so the path was what can we reuse? [13:03] What do we need to translate? What do we  need to post edit? Where do we need the [13:08] right quality checks? Where do we need human  review? And where can we deliver automatically? [13:17] And then the final layer is quality estimation [13:21] and risk routing. Risk routing being super  important for us, right? We we want to use LLM, [13:30] but we need to determine where there might  be risks within errors and error handling. [13:36] We absolutely cannot have pieces of content out  that have errors in there and so the detection [13:44] model is also very important. So the quality  score guides the decisions within this model. [13:51] Um and the essentially the business rules  are what help determine the outcome. [13:58] So terminology and formatting checks  need to be performed. We need to make [14:02] sure um everything is complete and consistent. So  there needs to be a validation and check there. [14:08] We need to do automated quality estimation  and not rely on our humans to do quality [14:14] estimation all the time. We need to do a model  comparison and anomaly detection within it. [14:23] And then something that might be unique to us  I think certainly is very important to us is [14:31] we actually do sampling and human loop feedbacks.  So what we do is it that sampling does not stop [14:39] the process but concurrent to this process we  take samples and just test that for our own data. [14:48] So that's that bottom portion of the  architecture you see which is score, [14:53] decide and deliver or route for review.  So if it's high confidence plus low risk, [15:01] we move that along. We deliver it automatically.  But if it's high risk, we move it to a human. [15:11] So, as I'm sure you've probably  seen in a lot of discussions [15:15] throughout this week at this conference, the human  role is changing as a whole. And that's certainly [15:21] no different within language services. So, human  expertise becomes the valuable piece that helps [15:30] shape this architecture as a whole. Right? So  we don't want people to do more manual routing, [15:41] chasing requesters, you know, moving things  through different avenues and handoffs [15:48] and correcting essentially  the same issues repetitively. [15:53] What we would like them to do now is help us  define these policies that go into these systems. [16:00] Review the expectations or the exceptions.  Sorry. Where are their exceptions? So we know the [16:07] the things that are meant are routed all the  time, but what types of content have exceptions [16:13] that could be reviewed because those exceptions  could be trained. And so that feedback test is [16:20] very valuable in order to not only fine-tune  this architecture but also retrain the LLMs. [16:31] So what does trusted automation look like within  language services at Indeed? It's a system [16:38] that knows what to automate, what not to  automate, but more importantly can tell [16:43] the difference between the two. So routine  content checks that can be done quickly. [16:50] We want to take critical content and  put stronger controls around those [16:55] and look at where expectations are  visible and explainable through data. [17:02] And another thing that we're able to do is  actually detect hallucinations through the scoring [17:09] and then the feedback that is constantly  provided to make future decisions. [17:15] So the takeaway here is yes AI is powerful. I  think we all know that. That's why we're all here. [17:21] But trust is what lets language  operations use it at scale. [17:31] So trust in AI can be built through an  architecture with the right level of human [17:37] effort and review. And that's really  the core of this architecture for us. [17:46] Thank you so much for your time  today. I hope you found this valuable.