## Background
Major League Soccer's Alireza Aliamiri delivered this presentation at Ai4, published on YouTube on August 16, 2026. According to the page description, the session explores how sports organizations can move beyond productivity tools toward production-grade agentic AI platforms across compliance, content production, and fan support while maintaining governance, observability, and human-in-the-loop controls.
## TL;DR
Alireza Aliamiri details Major League Soccer's strategy for transitioning AI from impressive isolated demos into governed, production-grade enterprise platforms. Using real-world applications like 'Recap AI'—which generates post-match summaries using 40 automated quality checks and human review escalations—he outlines how MLS balances automation with strict risk controls. The presentation highlights five core architectural pillars, including trusted context and smart model routers, to scale AI across high-stakes workflows like player contracts and media operations. Ultimately, Aliamiri emphasizes that success relies on bounded scopes, robust evaluation pipelines, and tracking operational metrics like token usage to drive long-term ROI.
## Atmosphere & style
The presentation carries a fast-paced, pragmatic, and executive tone, opening with a lighthearted promise to keep the talk quick and impactful 'like an espresso shot' before happy hour. Aliamiri speaks with the authority of an enterprise practitioner, grounding complex technical concepts in real operational lessons rather than speculative hype. The narrative moves smoothly from real-world problem statements to structured architectural frameworks, delivering an engaging and insightful session focused on enterprise reliability and trust.
## Key takeaways
- [00:04](https://www.youtube.com/watch?v=y2a6kgTBwxc&t=4s) Enterprise AI must transition from impressive controlled demos to production-grade systems capable of managing live data streams, edge cases, and API failures.
- [00:04](https://www.youtube.com/watch?v=y2a6kgTBwxc&t=4s) Major League Soccer (MLS) deploys AI across multiple operational streams, including content creation, contract operations, media monitoring, and marketing outreach.
- [00:04](https://www.youtube.com/watch?v=y2a6kgTBwxc&t=4s) Human-in-the-loop controls should be intentionally integrated based on outcome risk rather than added as an afterthought.
- [03:58](https://www.youtube.com/watch?v=y2a6kgTBwxc&t=238s) MLS's Recap AI generates fan match summaries by bounding model output to two or three structured sentences to eliminate hallucinated stats and timeline errors.
- [03:58](https://www.youtube.com/watch?v=y2a6kgTBwxc&t=238s) Recap AI employs a verification pipeline of 40 quality checks—including 32 critical checks that immediately route failing summaries to human review.
- [03:58](https://www.youtube.com/watch?v=y2a6kgTBwxc&t=238s) The Recap AI system utilizes a secondary LLM as an evaluator that allows up to three generation retries before escalating to human intervention.
- [07:55](https://www.youtube.com/watch?v=y2a6kgTBwxc&t=475s) Decision autonomy scales by risk level: low-stakes fan content moves toward automation, whereas high-stakes player contract operations enforce mandatory human sign-offs.
- [07:55](https://www.youtube.com/watch?v=y2a6kgTBwxc&t=475s) Bounding system scope prevents models from overreaching and ensures failure modes remain traceable and debuggable.
- [10:39](https://www.youtube.com/watch?v=y2a6kgTBwxc&t=639s) Production-ready AI platforms depend on five core pillars: Trusted Context, Orchestration, Model Choice, Controls & Validation, and Runtime Foundation.
- [10:39](https://www.youtube.com/watch?v=y2a6kgTBwxc&t=639s) Smart model routers allow enterprise platforms to dynamic dispatch tasks based on complexity, cost efficiency, and latency requirements rather than relying on a single LLM.
- [13:44](https://www.youtube.com/watch?v=y2a6kgTBwxc&t=824s) Human feedback serves as the system's outer loop, building long-term trust and aiding continuous platform improvement.
- [13:44](https://www.youtube.com/watch?v=y2a6kgTBwxc&t=824s) Monitoring token consumption spikes helps engineers identify redundant agentic reasoning chains and replace them with deterministic API tool calls.
## Detailed walkthrough
### [00:04](https://www.youtube.com/watch?v=y2a6kgTBwxc&t=4s) From Demos to Production in Enterprise AI
Alireza Aliamiri, Head of Engineering, Data, and AI at Major League Soccer (MLS), opens his presentation by acknowledging his position as one of the final speakers before the happy hour, promising to keep his talk quick and impactful "like an espresso shot." He introduces the core focus of the session: moving AI solutions from impressive initial demos to production-grade operational realities. Within MLS, AI is actively driving operational efficiencies across various streams, including content creation, player contract operations, media operations, and marketing outreach.
While demos created in tools like ChatGPT are inherently impressive to enterprise stakeholders when reviewed manually, production environments introduce live data streams, context nuances, system failures, API integration hurdles, and strict verification requirements. A smart AI model is only one component of a broader production system where surrounding elements can easily fail. Successful deployment requires clearly defining expected outcomes and operational metrics rather than treating reliability as an afterthought.
- Demos work well in controlled environments with human oversight, but production requires handling exception cases, evidence tracing, and tool integration.
- Human-in-the-loop controls must be integrated intentionally based on outcome risk rather than added after deployment.
---
[03:58](https://www.youtube.com/watch?v=y2a6kgTBwxc&t=238s) Case Study: MLS Recap AI and Governance Controls
Aliamiri details "Recap AI," a production system at MLS that generates post-match summaries for fans immediately after the final whistle by ingesting live scores, statistics, and game events. Early tests feeding raw match data into general LLMs yielded appealing news text, but deeper evaluation revealed hallucinated player names and broken chronological order. To solve this, MLS bounded the application's output to two or three highly structured sentences and built a rigorous verification pipeline combining deterministic software checks and a secondary LLM.
To guarantee output quality and accuracy, the Recap AI system runs 40 distinct quality checks on every generated summary before publication. These checks cover factual precision against official stats APIs, editorial tone compliance, and chronological event sequence. Among these, 32 are classified as critical checks; if a summary fails a single critical check, it enters exception handling and is routed to human review rather than being published automatically.
- System architecture flows across four distinct stages: data context understanding, retrieval and stats API integration, orchestration/generation, and human-in-the-loop publishing/escalation.
- The pipeline utilizes a powerful secondary LLM as a evaluator/judge that allows up to three generation retries before triggering human intervention.
---
[07:55](https://www.youtube.com/watch?v=y2a6kgTBwxc&t=475s) Design Patterns Across High-Stakes Sports Workflows
The design patterns established in Recap AI extend across several other MLS enterprise initiatives at varying levels of maturity. In player contract operations, AI assists by reading complex contract documentation and cross-checking terms against player performance metrics. Because contractual decisions represent high-stakes operations, human-in-the-loop approval is strictly enforced, keeping the ultimate decision-making power with human experts.
Similar architectural principles apply to computer vision systems monitoring visual compliance for player gear and uniforms during matches, as well as media operation tools that detect broadcast interruptions. Across all these use cases, the framework remains consistent: bounding the scope of operation, maintaining full evidence audit trails for trace-ability, and designing automated escalation paths when confidence checks fail.
- Decision autonomy exists on a spectrum: low-stakes fan content moves toward complete automation, while high-stakes contract operations mandate strict human sign-offs.
- Bounding system operations prevents models from attempting overbroad tasks, making failure modes traceable and debuggable.
---
[10:39](https://www.youtube.com/watch?v=y2a6kgTBwxc&t=639s) Core Architectural Pillars of Production AI Platforms
Transitioning intelligent AI models into production-ready operational systems depends on five essential structural pillars:
1. **Trusted Context**: Ensuring source data (such as match stats and scores) is pristine, timely, and verified before ingestion.
2. **Orchestration**: Breaking workflows down into modular retrieval steps, tool executions, and API calls to simplify debugging.
3. **Model Choice**: Selecting models that meet exact latency and performance targets for specific operational roles.
4. **Controls & Validation**: Embedding exception handling, validation checks, and audit trails directly into the framework.
5. **Runtime Foundation**: Leveraging robust cloud or on-premise infrastructure to maintain system availability.
Aliamiri highlights the emerging industry shift toward smart model routers. Rather than relying on a single LLM, enterprise platforms deploy routers to dispatch prompts to appropriate models based on task complexity, cost efficiency, and response latency requirements.
- Trust in autonomous enterprise systems is hard to earn and easy to lose, making reliable controls critical for user adoption.
- Framework modularity allows organizations to reuse core orchestration and governance infrastructure across drastically different business domains.
---
[13:44](https://www.youtube.com/watch?v=y2a6kgTBwxc&t=824s) Strategic Takeaways and Q&A on Metrics & Token Tracking
Aliamiri summarizes key recommendations for building operational AI workflows: begin with bounded workflows, establish evaluation frameworks prior to deployment, preserve evidence traces and human feedback loops to improve trust over time, bake escalation design directly into systems, and measure tangible performance metrics like publication speed and fan engagement to prove ROI.
During the Q&A session, an audience member asks about tracking non-traditional business metrics, such as fluctuating token expenditures and unexpected agent behaviors. Aliamiri explains that token usage tracking serves as a valuable tool for system optimization; exponential token growth often reveals that an agent is spending unnecessary cycles executing redundant code logic. Recognizing these patterns allows engineers to convert frequent reasoning chains into dedicated API tool calls, optimizing both cost and latency as usage scales across enterprise applications.
- "Human feedback is technically the outer loop of your system where you improve the trust of the system."
- Monitoring token consumption patterns helps identify opportunities to replace expensive agentic reasoning steps with deterministic API functions.
## Notable quotes
> [13:44](https://www.youtube.com/watch?v=y2a6kgTBwxc&t=824s) Human feedback is technically the outer loop of your system where you improve the trust of the system. — Alireza Aliamiri
## Mentioned
Alireza Aliamiri, Major League Soccer, MLS, ChatGPT, Recap AI
## Video description
Alireza Aliamiri - Head of Engineering, Data and A.I., Major League Soccer
Sports organizations are moving beyond AI as a personal productivity tool and toward agentic systems that operate across real enterprise workflows. Grounded in Major League Soccer’s experience, this session will explore how leagues and sports businesses can move from pilots and point solutions to production-grade agentic AI capabilities across contract and compliance, content understanding, generative production, personalization, and fan support.
We’ll examine what it takes to build an agentic AI platform for a modern sports organization, including orchestration across tools and data, governance, human-in-the-loop controls, observability, reliability, and ROI measurement tied to tangible efficiency gains and improved stakeholder experiences.
Attendees will leave with a practical framework for scaling AI agents from promising demos into governed platform capabilities that improve how sports organizations operate, support partners and teams, and engage fans.
Alireza Aliamiri, Head of Engineering, Data, and AI at Major League Soccer, outlines how enterprise sports organizations transition AI from isolated demos to dependable production platforms through bounded task scopes, multi-layered verification, and risk-adjusted human oversight.
Architectural Pillars for Live Enterprise Deployments (0:04)
Aliamiri emphasizes that while AI demonstrations in controlled tools like ChatGPT perform well with manual oversight, live enterprise operations face data stream fluctuations, API integration failures, and strict verification requirements. Because models represent only one component of a larger system where surrounding infrastructure can fail, reliability and operational metrics must be designed upfront rather than added retroactively.
To bridge this gap, MLS structures its production framework around five operational pillars: trusted context through verified source data, modular orchestration to simplify debugging, targeted model choice, embedded validation controls, and resilient runtime infrastructure. Additionally, enterprise deployments increasingly adopt smart model routers to dispatch tasks based on complexity, latency, and cost efficiency rather than relying on a single general model.
Multi-Layered Verification in Automated Match Recaps (3:58)
MLS’s "Recap AI" platform illustrates these production controls by generating post-match summaries immediately after the final whistle. Unconstrained early tests with general large language models generated convincing prose but suffered from factual hallucinations and broken chronological sequences. In response, MLS bounded the system's output to two or three structured sentences and introduced an automated verification pipeline.
The system runs 40 distinct quality checks on every summary against official statistics APIs and editorial guidelines. Of these, 32 are designated as critical. A secondary evaluator model allows up to three generation retries, but if a single critical check fails after retries, the output bypasses autonomous publication and routes directly to human exception handling.
Calibrating Autonomy to Operational Risk (7:55)
Across MLS, automation levels vary along a spectrum determined by operational stakes. While fan-facing match summaries shift toward full automation under automated quality gates, high-stakes domains strictly mandate human decision-making. In player contract operations, where AI cross-references contractual documentation with on-field performance metrics, human-in-the-loop review is mandatory before final decisions are made.
Similar design patterns apply to computer vision systems monitoring uniform compliance and media operations tracking broadcast interruptions. In every deployment, bounding the operational scope prevents models from tackling overbroad tasks, while end-to-end evidence audit trails preserve traceability when anomalies occur.
Token Tracking and Tooling Optimization (13:44)
Beyond traditional business metrics like publication speed and fan engagement, Aliamiri highlights token consumption tracking as an operational diagnostic. Rapid or exponential token growth often indicates that an agent is caught in redundant reasoning loops or executing excessive internal code logic.
By identifying these recurring reasoning chains, engineering teams can convert expensive, dynamic agentic deliberation into fast, deterministic API tool calls. Aliamiri views ongoing human feedback as an outer loop that maintains trust and guides system refinement as workloads scale across the enterprise.
MLS's production approach demonstrates that deploying agentic AI in enterprise settings requires replacing unconstrained model prompts with bounded tasks, strict programmatic validation, and telemetry that flags when agent reasoning should be replaced by deterministic software.
精炼阅读摘要
美国职业足球大联盟（MLS）工程、数据与人工智能主管 Alireza Aliamiri 指出，将企业级人工智能从演示原型推进至生产环境，必须摆脱对通用大语言模型直接输出的盲目依赖，转而构建包含严格范围限定、多层确定性校验以及按风险等级划分的人机协同治理体系。
从演示原型迈向生产级系统 (0:04)
在 ChatGPT 等工具中构建的原型演示虽然能够在受控环境及人工监督下展现出令人信服的效果，但企业真实生产环境却充满了实时数据流、复杂的上下文语义、突发系统故障、API 集成壁垒以及严格的验证要求。Aliamiri 强调，智能大语言模型仅仅是庞大生产系统中的一个环节，围绕模型运行的外部组件极易出现故障。企业若要在生产中取得稳定成果，必须在立项之初就明确界定预期结果与具体的运营指标，而不能将系统的可靠性视为事后补救的附加项。
为此，MLS 采取的关键工程策略之一是对系统任务进行严格的范围界定（bounding）。在处理赛事数据时，早期测试表明，若直接将原始比赛数据输入通用大模型，生成的文本虽具可读性，却频繁出现虚构球员姓名或颠倒事件时间线等幻觉问题。MLS 最终将生成任务严格约束在两至三句高度结构化的句子内，通过收缩输出空间，使得系统潜在的故障模式变得清晰、可追溯且易于调试。
Recap AI 的多重校验与异常处理 (3:58)
以 MLS 的“Recap AI”系统为例，该系统旨在比赛终场哨声响起后，立即整合比分、统计数据与关键事件，自动为球迷生成赛后总结。其技术流程横跨四个明确阶段：数据上下文理解、检索与统计 API 集成、任务编排与生成、以及人机协同发布与升级机制。为确保输出内容的事实准确性与语调规范，系统在正式发布前会对每篇生成的总结执行多达 40 项独立的质量检查，全方位核对官方统计 API 的数据精度、编辑语气及事件的时间逻辑。
在所有 40 项检查中，有 32 项被设立为关键检查项。整个校验流水线部署了能力更强的次级 LLM 作为评估裁判，在触发人工干预前允许系统进行最多三次重新生成尝试；然而，一旦生成结果未能通过哪怕一项关键检查，系统便会强制中断自动发布流程，将其转入异常处理队列并交由人工进行审核，坚决杜绝带有事实缺陷的内容直接呈现在受众面前。
基于风险等级的人机协同与场景复用 (7:55)
Recap AI 验证成熟的架构模式正在被推广至 MLS 内部其他处于不同发展阶段的业务板块中，且系统在各环节的决策自主权完全取决于业务本身的风险严重程度。对于面向球迷的赛后简讯等低风险场景，系统正稳步向端到端的全自动化演进；但在球员合同运营等高风险任务中，尽管 AI 能够协助审阅复杂的合同条款并与球员表现指标进行跨维度核对，最终的决策与签署权限依然被强制保留在人类专家手中。
同样的工程原则亦被贯彻至其他高要求场景，例如利用计算机视觉系统监控比赛现场球员球衣与装备的合规性，以及监测媒体运营中广播信号的中断异常。Aliamiri 解释称，无论具体业务如何变化，其底层框架始终保持高度一致：限定操作边界、建立完整的证据审计追踪链条以备核查，并在置信度检查失败时提供确定性的自动升级通道。
平台核心支柱与推理成本工程优化 (10:39)
Aliamiri 归纳指出，构建生产级智能体 AI 平台必须依托五大架构支柱：受信任的上下文（确保原始数据在摄入前高度准确、及时且经过校验）、模块化编排机制（将流程拆解为独立的检索、工具执行与 API 调用以便排错）、针对延迟与性能匹配的模型选型、内嵌的控制与验证机制，以及稳固的运行时基础设施。为进一步优化效率，MLS 开始采用智能模型路由器，依据具体任务的逻辑复杂度、响应延迟和资金成本，将请求分发给最合适的模型，而非全盘依赖单一 LLM。
在日常运维层面，监控非传统业务指标（如代币 token 消耗趋势）对优化系统性能至关重要。Aliamiri 透露，智能体代币消耗出现指数级暴涨往往暴露出模型正在执行冗余的代码逻辑或死循环；工程团队通过捕获这些异常的推理链模式，能够将高频出现的认知推理步骤改写为确定性的专用 API 工具调用，从而在平台业务扩张的同时显著缩短响应延迟并压低计算成本。
MLS 的实践表明，企业级 AI 的信任建立是一个长期迭代的过程，必须将人类反馈设计为系统的外环回路，以持续加固系统的安全边界。尽管通过范围约束、次级模型评审和 API 化改造能够有效解决幻觉与算力冗余问题，但在面对多模态实时数据与高风险商业决策时，如何动态平衡机器自主性与人工干预成本，仍是企业级智能体平台需要持续验证的核心课题。
Detailed notes — zh
## Background
该视频由 Ai4 频道于 2026 年 8 月 16 日发布在 YouTube 平台上，记录了 MLS（美国职业足球大联盟）工程、数据与 AI 负责人 Alireza Aliamiri 的主题演讲。视频主要探讨了体育组织如何超越简单的 AI 个人生产力工具，构建涵盖合同履约、内容理解、生成式生产和球迷互动的生产级 Agentic AI 平台。
## TL;DR
演讲围绕 MLS 如何将 AI 从令人惊艳的 Demo 推进到高可用的生产环境展开。Aliamiri 以赛后总结产品 Recap AI 为例，详细拆解了由数据检索、编排、发布及异常处理组成的四大核心架构，并展示了包含 40 项质量检查（32 项关键校验）和最多 3 轮自动重试的严格把关机制。针对球员合同等高风险场景，系统强制将人工干预与证据追溯嵌入底座，同时通过模型路由与 API 封装来控制 Token 指数级增长的成本。最后，他总结了从限定边界切入、评测先行到内置升级机制等五条落地金科玉律。
## Atmosphere & style
整场演讲节奏明快、干货满满，Aliamiri 自诩演讲如同一杯“意式浓缩咖啡”般短小精悍且充满冲击力。他以幽默风趣的台风拉开序幕，随后迅速切入硬核的技术架构与落地痛点，将复杂的 AI 治理与工程实践讲得条理清晰、深入浅出。演讲兼具商业洞察与技术实践的务实感，既有无人驾驶汽车比喻带来的警示感，也有面对生产线“奇葩异常”时的经验谈，为观众带来了兼具启发性与实操性的现场体验。
## Key takeaways
- [00:04](https://www.youtube.com/watch?v=y2a6kgTBwxc&t=4s) 从令人惊艳的 Demo 走向生产环境（Production）面临巨大鸿沟，生产级 AI 必须实时面对动态数据、失败异常、事实校验及证据链留存等硬性挑战。
- [00:04](https://www.youtube.com/watch?v=y2a6kgTBwxc&t=4s) MLS 将 AI 应用于内容创作、球员合同履约与管理、媒体运营以及市场推广等多个核心业务环节，显著提升组织运营效率。
- [05:58](https://www.youtube.com/watch?v=y2a6kgTBwxc&t=358s) MLS 推出 Recap AI 赛后自动生成摘要产品，将输出字数严格限制在 2 至 3 句话以内，以兼顾结构化校验与球迷快速阅读需求。
- [05:58](https://www.youtube.com/watch?v=y2a6kgTBwxc&t=358s) Recap AI 在摘要生成后会执行多达 40 项质量检查（包含 32 项致命校验规则），确保事实准确、时间线顺规及符合编辑风格。
- [05:58](https://www.youtube.com/watch?v=y2a6kgTBwxc&t=358s) 质量校验失败时，系统最多允许调用高阶 LLM 自动化重试 3 轮，若仍无法通过则强制进入人工复核流程。
- [07:55](https://www.youtube.com/watch?v=y2a6kgTBwxc&t=475s) 生产级 AI 架构包含四大核心阶段：数据检索与 API 集成、编排与作业执行、发布与结果呈现，以及异常处理与人工干预。
- [07:55](https://www.youtube.com/watch?v=y2a6kgTBwxc&t=475s) AI 自治程度与业务风险成反比：高风险场景（如球员合同履约）必须强制设定人工审核为最终决策节点并具备可追溯性。
- [10:39](https://www.youtube.com/watch?v=y2a6kgTBwxc&t=639s) 生产级 AI 系统的四大柱石包括可信上下文（Trusted Context）、模块化编排（Orchestration）、模型路由（Model Choice & Router）以及控制与运维（Controls & Operations）。
- [10:39](https://www.youtube.com/watch?v=y2a6kgTBwxc&t=639s) 建立生产级 AI 平台需要明确控制与监控，正如无人驾驶汽车一样，一次不可靠的结果就会彻底摧毁用户建立的信任。
- [13:44](https://www.youtube.com/watch?v=y2a6kgTBwxc&t=824s) 企业构建 AI 工作流的五大原则：边界清晰（Start bounded）、评测先行、留存证据、内置人工升级机制以及量化 ROI。
- [13:44](https://www.youtube.com/watch?v=y2a6kgTBwxc&t=824s) 系统 Agent 化程度越高，Token 消耗量呈现指数级增长，需要通过 Token 日志监控来发现并优化低效推理循环。
- [13:44](https://www.youtube.com/watch?v=y2a6kgTBwxc&t=824s) 针对频繁消耗 Token 的推理步骤，可以通过抽离并封装为传统 API / Tool Call 的方式来实现降本增效。
## Detailed walkthrough
### [00:04](https://www.youtube.com/watch?v=y2a6kgTBwxc&t=4s) 从 Demo 走向 Production 的核心挑战
MLS（美国职业足球大联盟）工程、数据与 AI 负责人 Reza Aliamiri 登台发言。作为 Happy Hour 前的最后几位演讲者之一，他幽默地表示自己的演讲将像一杯意式浓缩咖啡（Espresso Shot）一样“短小精悍且充满冲击力”。在 MLS，AI 已经深入到内容创作、球员合同履约与管理、媒体运营以及市场推广等多个业务环节，极大地提升了组织内部的运营效率。
然而，将 AI 从令人惊艳的 Demo（演示 demo）推进到真正的生产环境（Production）却面临巨大鸿沟。直接在 ChatGPT 中输入 prompt 获取炫酷结果非常容易，且有内部人员在侧监视；但一旦部署至生产线，系统就必须实时面对动态数据接入、各种失败异常、复杂的上下文环境、事实校验以及证据链留存等硬性挑战。这需要团队清晰定义预期成果与关键运营指标，将人工干预（Human-in-the-loop）作为核心架构设计，而非事后补救措施。
- 演讲者：Alireza Aliamiri（MLS 工程、数据与 AI 负责人）
- MLS 内部应用 AI 提升运营效率的主要领域：内容（Content）、球员合同运营（Player contract operations）、媒体运营（Media operations）、市场营销（Marketing outreach）
- 生产环境的关键要素：实时数据接入（Live data）、失败容错（Failures）、上下文管理（Context）、结果验证（Verification）、证据留存（Evidence）、异常处理（Exception handling）
---
### [03:58](https://www.youtube.com/watch?v=y2a6kgTBwxc&t=238s) Recap AI 落地实践与 40 道质量把关机制
为了说明如何将 AI 落地到生产环境，Aliamiri 详细拆解了 MLS 推出的一款名为 **Recap AI** 的产品。该产品旨在完场哨声响起后，自动整合比赛比分、球员统计等实时数据，为球迷快速生成简明扼要的赛后总结。虽然直接将新闻素材丢给通用大模型也能生成文采斐然的报道，但细究之下就会暴露各种问题，例如时间线错乱、幻觉以及球员姓名拼错等。
为了解决这一痛点，团队采取了“限定边界与严格校验”的策略：将 Recap AI 的输出字数严格限制在 2 至 3 句话以内，并建立了一个多阶段的流水线架构。整个流程包含理解与上下文触发、数据检索与 API 对接（调用 Stats API 提取球员和比赛数据）、生成，以及最为关键的**验证与质量检测**阶段。
```
[比赛结束数据] ──> [检索与 API 集成 (Stats API)] ──> [LLM 生成摘要]
│
▼
[异常处理 / 人工审核] <── (校验失败) ─── [40 项质量检查 (32 项关键规约)]
│ │
└── (通过校验) ─────────────────────────────┴──> [自动发布给球迷]
```
在摘要生成后，系统会自动运行多达 **40 项质量检查**，其中包括 32 项极为关键的致命校验规则（Critical checks）。检查范围涵盖事实准确性、时间线顺规（避免先讲比赛结尾再讲开头）以及符合 MLS 编辑风格的语言基调（Editorial tone）。如果有任何一项关键校验失败，系统会触发重试逻辑（调用更强大的 LLM 进行三轮自动重试）或直接丢入异常处理通道，交由人工编辑复核，绝不盲目发布。
- 关键指标：Recap AI 在摘要生成后会执行 **40 项质量检查**，其中包含 **32 项关键（Critical）校验**
- 最多允许调用高阶 LLM 自动化重试 **3 轮**，若仍未通过则强制进入人工复核流程
- 摘要输出限制：严格控制在 **2-3 句话**，既方便结构化校验，又符合球迷快速阅读的需求
---
### [07:55](https://www.youtube.com/watch?v=y2a6kgTBwxc&t=475s) 生产级 Agentic AI 架构模式与高风险决策的治理
通过 Recap AI 的落地，MLS 总结出了一套可复用的生产级 AI 架构模式，主要涵盖四个核心阶段：**数据检索与 API 集成（Retrieval & Integration）**、**编排与作业执行（Orchestration & Work）**、**发布与结果呈现（Publishing）**，以及**异常处理与人工干预（Exception Handling & Human-in-the-loop）**。
这套通用框架被成功复用到 MLS 其他不同成熟度的 AI 项目中。例如在**球员合同运营（Player Contract Operations）**场景中，AI 用于解析复杂的合同条款并比对球员的实际表现，但这属于极高风险（High-stakes）的业务决策，因此人工审核被强制设为最终决策节点；在**视觉合规审查（Visual Compliance）**项目中，利用计算机视觉技术识别球员身穿的官方赞助装备；在**媒体运营（Media Operations）**中，AI 能在直播信号中断时快速感知并定位问题，但最终调度资源仍由人类决定。
Aliamiri 强调，自治程度与风险等级成反比：对于 Recap AI 这种低风险内容，在系统建立足够信任度后可逐步走向完全自治；而涉及法律、合同等高风险场景，必须将“证据追踪”（Audit trails / Traceability）和“人工升级机制”嵌入到系统骨架之中。
```
+-----------------------------------------------------------------------+
| AI 决策自治度与风险光谱 |
+-----------------------------------------------------------------------+
[高自治 / 低风险] [低自治 / 高风险]
Recap AI 赛后总结 ────────> 视觉合规审查 ────────> 球员合同与法律履约
(自动化发布/高容错) (计算机视觉辅助) (强制 Human-in-the-loop)
+-----------------------------------------------------------------------+
```
- 核心架构四大模块：检索集成、编编调度、发布呈现、异常与人工干预
- 关键原则：系统必须具备**可追溯性（Evidence / Tracing）**，确保每项 AI 决策都有据可查，且异常升级路线必须在系统设计之初就完成构筑，而非事后补丁
---
### [10:39](https://www.youtube.com/watch?v=y2a6kgTBwxc&t=639s) 生产级 AI 系统的关键支撑柱与模型路由设计
要将智能 AI 模型真正融入企业生产环境，Aliamiri 归纳了几大不可或缺的支撑柱（Pillars）。首先是**可信上下文（Trusted Context）**：输入给 AI 的数据源（如比分、统计数据）必须极为精准且按时到达。其次是**模块化编排（Orchestration）**：将任务拆解为可调用的 API 工具，便于后期调试与维护。再者是**模型选择与路由机制（Model Choice & Router System）**：企业不应依赖单一模型，而应引入智能路由（Router），根据成本、延迟与任务复杂度动态分发请求。
此外，**安全控制与监控（Controls & Operational Monitoring）**是赢得用户信任的关键所在。 Ali Amiri 引用上一位演讲者关于无人驾驶汽车的经典比喻：无人驾驶汽车即使安全行驶了数百万英里，只要发生一次严重事故，人们建立的信任就会彻底崩塌；自治 AI 系统同样如此，一旦输出不可靠的结果，企业内部或用户的信任将难以挽回。
- 生产级 AI 框架四大柱石：
1. **Trusted Context（可信上下文）**：确保源头数据的时效性与绝对准确
2. **Orchestration（模块化编排）**：便于调试与 API 工具化解耦
3. **Model Choice & Router（模型路由）**：按延迟、成本与能力动态分发模型请求
4. **Controls & Operations（控制与运维）**：包含校验、追溯、异常升级与监控
---
### [13:44](https://www.youtube.com/watch?v=y2a6kgTBwxc&t=824s) 落地核心经验总结与 Q&A：应对边缘异常与 Token 成本
演讲最后，Aliamiri 给出了构建企业级 AI 运营工作流的五条金科玉律：
1. **边界清晰（Start bounded）**：从限定范围的小工作流切入，切忌“把大海煮沸”（Boil the ocean）。
2. **评测先行（Define evaluation before deployment）**：在部署上线前就确定好评估标准。
3. **留存证据（Preserve evidence）**：同时留存 AI 决策链路与人工反馈（Human feedback）数据，形成闭环优化。
4. **内置升级（Design escalation into the system）**：将人类接管机制直接写入系统底层逻辑。
5. **量化 ROI（Measure ROI）**：明确衡量业务指标（如球迷参与度提升、发布时效加快等）。
在随后的问答环节中，观众提问：“除了常规业务 KPI，如何应对像 Token 消耗这种新变量，以及如何处理那些意想不到的奇葩异常（Weird exceptions）？”
Aliamiri 回答道，Token 成本追踪其实是系统优化的重要抓手。MLS 的实践表明，当系统 Agent 化（Agentic）程度越高时，Token 消耗量会呈指数级增长。通过分析 Token 监控日志，他们常能发现系统在某些重复代码上浪费了大量 Token，此时只需将该环节抽离并封装为一个独立的 API 工具供 Agent 调用，就能瞬间实现极大幅度的降本增效。面对复杂的生产环境，唯有持续追踪与针对性优化，才能建立起真正高可用、高信任的生产级 AI 平台。
- 业务价值（ROI）衡量点：球迷参与度（Fan engagement）、新闻发布速度（Publish speed）
- Agent 架构的 Token 特性：Agent 化程度越高，Token 消耗呈**指数级增长**（Exponential growth）
- 优化技巧：利用 Token 跟踪定位低效循环，将频繁消耗 Token 的推理步骤重构为传统的 **API / Tool Call**，以优化延迟与成本
## Notable quotes
> [00:04](https://www.youtube.com/watch?v=y2a6kgTBwxc&t=4s) 像一杯意式浓缩咖啡（Espresso Shot）一样“短小精悍且充满冲击力” — Alireza Aliamiri
> [00:04](https://www.youtube.com/watch?v=y2a6kgTBwxc&t=4s) 将人工干预（Human-in-the-loop）作为核心架构设计，而非事后补救措施 — Alireza Aliamiri
> [03:58](https://www.youtube.com/watch?v=y2a6kgTBwxc&t=238s) 限定边界与严格校验 — Alireza Aliamiri
> [07:55](https://www.youtube.com/watch?v=y2a6kgTBwxc&t=475s) 自治程度与风险等级成反比 — Alireza Aliamiri
> [07:55](https://www.youtube.com/watch?v=y2a6kgTBwxc&t=475s) 必须将“证据追踪”（Audit trails / Traceability）和“人工升级机制”嵌入到系统骨架之中 — Alireza Aliamiri
> [10:39](https://www.youtube.com/watch?v=y2a6kgTBwxc&t=639s) 无人驾驶汽车即使安全行驶了数百万英里，只要发生一次严重事故，人们建立的信任就会彻底崩塌 — Alireza Aliamiri
> [13:44](https://www.youtube.com/watch?v=y2a6kgTBwxc&t=824s) 切忌“把大海煮沸”（Boil the ocean） — Alireza Aliamiri
> [13:44](https://www.youtube.com/watch?v=y2a6kgTBwxc&t=824s) 当系统 Agent 化（Agentic）程度越高时，Token 消耗量会呈指数级增长 — Alireza Aliamiri
## Mentioned
MLS, Major League Soccer, Alireza Aliamiri, Reza Aliamiri, Ai4, Recap AI, Stats API, ChatGPT, YouTube
## Video description
Alireza Aliamiri - Head of Engineering, Data and A.I., Major League Soccer
Sports organizations are moving beyond AI as a personal productivity tool and toward agentic systems that operate across real enterprise workflows. Grounded in Major League Soccer’s experience, this session will explore how leagues and sports businesses can move from pilots and point solutions to production-grade agentic AI capabilities across contract and compliance, content understanding, generative production, personalization, and fan support.
We’ll examine what it takes to build an agentic AI platform for a modern sports organization, including orchestration across tools and data, governance, human-in-the-loop controls, observability, reliability, and ROI measurement tied to tangible efficiency gains and improved stakeholder experiences.
Attendees will leave with a practical framework for scaling AI agents from promising demos into governed platform capabilities that improve how sports organizations operate, support partners and teams, and engage fans.
[Source](https://www.youtube.com/watch?v=y2a6kgTBwxc)
[13:44] Alireza Aliamiri: Human feedback is technically the outer loop of your system where you improve the trust of the system.
Original transcript
[0:04] Hello everyone. My name is Reza Ali Amiri and
I I lead the engineering data and AI in MLS.
[0:13] I know I'm the few last speakers before the happy
hours I'll make it short like espresso shot so
[0:19] quick and impactful. Um Today I want
to talk about um uh making AI solutions
[0:30] how to take it from the demo to production and
at MLS we have a lot of work work stream related
[0:37] to AI which AI is helping us with operational
efficiencies and you know the domains are content
[0:45] um player contract operations media operations
marketing outreach and all of those are you know
[0:54] domains where AI is helping us there initiatives
going on and I'm going to walk you through about
[0:59] you know our journey about like how we are taking
this from demo to production. I'll talk about um
[1:05] about first of all like how the demos are awesome
usually and then when you go to production there's
[1:10] a lot of you know stuff you have to put there to
make it work um and you know the challenge with
[1:17] live data with integration with tools and
cetera and essentially talking about that gap.
[1:26] And um also I'm going to show you a couple
of examples including a match recap where we
[1:32] we launched it in production a few
months ago and talk about what what
[1:38] some sort of like um capabilities we realize
or design patterns emerges from like taking
[1:45] something from demo to production. Um and the
the examples are related sports but you know
[1:53] I'm sure everyone would resonate with it you know
that the use cases might be different but I think
[1:57] the capabilities might might stay the same. Um
let me first talk about uh the uh the demos.
[2:05] Demos are awesome, right? I'm sure everyone in
the in the in your company or enterprise come to
[2:09] you with a nice demo, putting something in chat
GPT and it's like impressive, and production is
[2:15] very different. And I think most of the talks even
before me, the the speaker was talking about how
[2:21] uh things become more challenging. Um and the the
the challenge with with when you go to production
[2:28] with the Let me first talk about the the the
just the basic demo, right? You put something in
[2:32] the chat bot, you get some cool demo out of it.
There's a human looking at it, see the results.
[2:38] It's impressive. When you go to production,
you take that to production, there's live data,
[2:42] there's failures, there's context, there's
verification, um there is evidence has to be
[2:48] there, and um you know, exception handling and a
lot of productions that you have to think about.
[2:53] And um you can have a very smart model where
works perfectly in in in in your demo where good
[3:00] to production. That's part of the system, right?
So, you can have a lot of stuff that can go wrong
[3:05] around it. And um you know, what what we realize
is like you have to define the outcome clearly,
[3:11] right? What you're expecting from the system and
what your operational um you know, metrics are. Um
[3:18] the the other thing about the production is
there's human reviews. I think, you know,
[3:22] the previous speaker past speakers talked about
that about how you have human in loop, right?
[3:28] Uh versus having an agent like autonomous
human running, right? Especially when
[3:32] you have high stakes, you know, and the the
decision where you put the human in loop,
[3:37] where to put it and how much you want to afford
like um you know, enforce it is related to your
[3:42] outcomes, right? If you're dealing with a very
mission-critical like decisions, you want to have
[3:47] that there and is not an afterthought. So, you
have to have that um in in um in mind. Um Um so
[3:58] I'll talk about um what we learned from a recap
AI, which is a is a product we launched where
[4:07] after the final whistle, there's information
coming to us about the match, scores, stats.
[4:12] And the idea was we can create
something summary for the for the
[4:18] fans to quickly get what the match was about,
right? Um and of course, you know, when you
[4:23] feed it to a chat GPT with some news and stuff,
it creates a very, you know, appealing news.
[4:29] Um but when you deep deep dive into it, you see
like, you know, some cracks opening up about like
[4:34] the temporal components or, you know, some
hallucinations, player names and stuff. So
[4:40] what we did here is first of all, we we bounded
outcome, right? We we we said, "Okay, let's put
[4:46] something in production where it's we can verify
it." Um you know, to the deterministic, you know,
[4:52] rules. We can verify it with a more powerful LLM.
And also, we can have a human loop to verify the
[4:59] results. And if you can see, this has resemblance
to previous processes we are talking about. You
[5:05] have a a couple of stages. One is understanding,
your context come to you, data comes to you after
[5:10] the match. There's like, you know, software
around it that basically triggers the LLM,
[5:15] provides the data. And then there is a retrieval
and integration, API integrations are our data
[5:23] from the match recap come from stats API from the
the sources that, you know, stats of the players,
[5:30] stats of the match, and those are all come to the
LLM. And then the match would be generated. Um now
[5:38] the the important part of this thing is we limit
the output to two three sentences, right? And
[5:46] uh make it make we make sure we can verify it.
Now, this verification can be deterministic
[5:52] uh as well as LLM, right? Because we want to
make sure the system does it right. Um and
[5:59] this simple system by itself to get it to
production has 40 quality checks after the the
[6:05] the recap is generated, right? Um a lot
of them has to do with uh factual checks,
[6:10] right? You can do that quickly because it's
two three sentences, it's very structured.
[6:15] You can even do it programmatically. Um the
tone of editorial, right? Editorial will have
[6:20] a specific kind of format, so fans like it. Um
you know, factual information about chronological
[6:26] of events of the match. You don't want to talk
about the end of the match and talk about the
[6:29] first of the match or events back-to-back. Those
are all get factually checked and basically if any
[6:36] of them fail, 32 of them are critical, right? Any
of them fail, this goes to exception handling and
[6:42] you know, goes for human review, so we don't
publish it, right? This is very important. Um
[6:47] and um so if you see there's there's a lot
going on about permissions, validation,
[6:53] recovery, sometimes you can't see um the data
might not be available at that time, right? So,
[6:59] you have to retry. Um and the the tone of the
recap sometimes have to be adjusted, right? So,
[7:06] there's a more powerful LLM running
on top of the output that basically
[7:12] uh would trigger the retry, right? So, the
three retries goes on there and after that,
[7:17] you know, the the the powerful LLM as a judge
make a decision that this is good, right?
[7:22] And any of those failures would trigger a human
loop and um it wouldn't be published, right? So,
[7:28] if you see this example, um the reason I brought
it up is you see the a pattern emerging, right?
[7:35] Uh you might have different use cases, but um the
the framework and capabilities might be the same,
[7:40] right? So, there's there's four four
stages here, retrieval, integration,
[7:46] orchestration, and doing the work, publishing,
exception handling, and human loop, right? Um
[7:55] you'll see the same thing, for example,
the work we are doing with um with um
[8:01] with other stuff, you know, for example, we have
Recap AI, which is in production right now, and
[8:07] these are all different maturity levels, right? Um
we have player contract um operations where AI can
[8:14] read a a contract of a player, uh cross-check
it with uh their performance, and make sure,
[8:21] you know, they're they're check marks, right? Um
and that's a very high-stakes task, right? So,
[8:27] you want to put the human loop definitely there.
Like, the human is the one who makes the decision,
[8:31] right? You can't just run it autonomously. And
where do you put that is based on the, you know,
[8:35] impact of outcome. Um there's also, you know,
projects related to visual compliance, you know,
[8:42] certain, you know, sports gear that, you know,
player might might might wear. So, that's computer
[8:46] vision, right? But it's almost the same thing,
you know, you see the the patterns emerging,
[8:51] production issues, right? Uh this is operation,
right? You might have a media, you know,
[8:56] um interruption during the live match,
and AI can read it, can understand it,
[9:01] but human at the end of it would make a decision
about, you know, where to put the effort on.
[9:08] So, having said that, um you know, I I already
mentioned about, you know, where do you want to
[9:15] basically make your decisions, um and that would
be based on the the impact of the work, right? So,
[9:23] um I put the very basic, like, you
know, chart here in terms of, like,
[9:27] what is the stake, you know, what is at stake
in terms of, like, having fully autonomous work
[9:32] versus, like, having human in loop, right? For
example, Recap AI is, you know, in in in the
[9:39] left of the spectrum. Um at some point, we might
even let it go by itself, you know, like let it
[9:46] let it publish articles after you have confidence
in the system. Um you know, audit trails and um
[9:53] you know, human loop gets more important
when you're making decisions about like,
[9:57] you know, contractual decisions and everything,
right? So, it's important when you make the
[10:01] make the call and um one more thing I want to add
that it's it's very important to bound your um um
[10:08] bound your operations, right? Like what exactly
the system is doing is very confined. You're not
[10:13] trying to solve everything and monitor it, right?
Evidence is all through the system, right? So,
[10:19] any failure in the system, you have evidence of
it. So, tracking the the basically the traces
[10:24] of the decision-making is very important.
Um and making sure the exception handling
[10:30] and escalation to human is is designed by the
by the in your system versus an afterthought.
[10:39] So, um I mentioned about uh you know, what what
we learned um through these use cases and I think
[10:46] this applies to generally AI workflows, you
know, how you want to make an intelligent
[10:51] system like an AI models and merge it into a
production ready operation system, right? Um
[10:58] it has to have a couple of things, right? It has
to have trusted context, right? In in our case,
[11:03] it was Recap AI's like my data, scores, and stuff.
They have to be pristine, right? They have to make
[11:08] sure you're not feeding wrong data there, right?
And data should come on time and all this stuff,
[11:13] right? Um the other thing is orchestration, right?
So, the system would retrieve that information,
[11:19] make decision, do some work, make API calls, and
you break it down this way so you can modularize
[11:25] it and debug it. Um Third part is model choice,
right? The model choice here is important because
[11:32] um you want to make sure you have a model that
is performance. It works with your latencies and
[11:39] and also part of the output of the system,
right? You know, you can have a very smart model,
[11:45] but if you don't do the the surrounding
work, the the system would fail.
[11:50] Um you know, a couple of more related to this
pillars are controls, right? You need validation,
[12:00] you need exception handling, you need trace
and you know, in operations, right? So,
[12:07] those are socially baked into the framework
and this framework basically allows you to
[12:14] have a different use cases, but leverage the
same framework. And I feel a lot of enterprises
[12:19] are going that direction where you're breaking
down into the systems and you have monitoring
[12:24] and making sure you have a system that can run
in production with a trusted output. That's very
[12:31] important, you know, like you build a system,
you want to make sure people trust the output,
[12:35] right? And that's very hard hard to earn. The
previous speaker was talking about self-driving
[12:41] cars, right? You know, one accident with them
it it basically loses the trust, you know,
[12:46] versus like they can drive millions of miles and
usually with autonomous system you have to earn
[12:52] the trust making sure it works. Especially
in the high high high stake task. Um so
[13:01] And one more thing I want to mention is the
model selection. I think I talked about it.
[13:08] The other thing that is emerging in this
in this work operational workflows is,
[13:14] you know, not all models are the same. So,
I think what what would emerge here is like
[13:19] a router system, which is already I think a
lot of enterprises are using it to dispatch the
[13:25] dispatch the input and leverage the right model
here, and depending on the cost efficiencies and
[13:31] latencies, and you can leverage that.
Um and essentially, and then you have
[13:37] a runtime foundation which your cloud or
on-prem or whatever platform you're using.
[13:44] So, I'll wrap up with a couple
of takeaways from like building,
[13:49] um you know, an operational workflow for
AI system. Um number one is start with,
[13:56] you know, a bounded workflow, right? Making sure
you define the outcomes, what the system does,
[14:02] um in a in a limited fashion versus just boiling
the ocean. Um define evaluation before deployment.
[14:10] Um and preserve the the you know, the
evidence, you know, both from the AI
[14:17] as well as human feedback. Human feedback
is technically the outer loop of your system
[14:22] where you improve the the trust of the system,
right? Um and then design the escalation um
[14:30] in the system, bake it in the system, you know,
not an afterthought at when the system breaks,
[14:34] right? That's very important to do,
especially with high-stake tasks. And
[14:40] um last but not least is measure the performance,
right? Uh for example, if you're doing something
[14:46] related to I'll make an example with our uh recap
stories, right? Um how much is improved our,
[14:53] you know, fan engagement? Um how much that help
us to publish faster? How much engagement has
[14:58] gone up, right? I think those are like very
basic stuff we can measure, and make sure we
[15:03] get the ROI out of the AI systems and as we put
them in the operational workflows. Thank you.
[15:13] I guess how do you figure out what to track? Like
I assume you you start with existing workflows
[15:16] that you have like known business KPIs, start from
there, and I I sometimes I guess tokens are are
[15:23] themselves a new thing to track that every finance
department is is getting used to, but they're
[15:28] using using agents for things, I feel like you
can find the output falls into categories you
[15:33] didn't know to track. That weird weird things
can happen. Like, how do you lastly set up for
[15:40] it for lack of a better word, weird exceptions
that aren't normal KPIs to track. Yeah, um look,
[15:45] it's an ongoing thing, you know, the the token
cost is kind of like a little bit like tricky
[15:50] right now. They're they're changing, but
that's a great point. I think, you know,
[15:53] the the more you be able to track, you understand
how your system is performing, right? Um our
[15:59] experience has been like the more agentic work you
do, of course the the tokens exponentially grow.
[16:04] Um and sometimes you might look back and with
the token tracking figure out like your system
[16:09] is spending a lot of time spending on some,
you know, redundant code, right? And that would
[16:14] immediately become your API, right? You will build
a tool for it so you can make the tool car, right?
[16:19] I think those are really good optimization for,
I would say, cost optimization, latencies. Um you
[16:25] know, and I think based on the based on the use
case, you can define your own. Um and, you know,
[16:30] the use cases vary a lot, right? So, you can have
like sometimes you want a super like low latency
[16:36] system. You sometimes you want a super reliable
like, maybe you're making medical decision there,
[16:41] right? So, I think that that goes back uh to just
giving it thought um and, you know, tracking that
[16:47] and making sure you you put in metrics there. Um
as I said, like, you know, it's it's very easy
[16:51] to put something in ChatGPT and get some cool
results, but I think when you go to production,
[16:56] those are become really important, especially when
you have a lot of users, you have a lot of usage,
[17:00] um and they would they would basically amplify,
right? So, I think um that's a really good point.
