Tinder Deploys Fine-Tuned Small Models to Slash Moderation Latency

Making Generative Models Work in the Real World: From LLMs to SLMs With Tinder — Ai4

Ai4

Deploying generalist LLMs in production creates unsustainable latency and cost bottlenecks, requiring specialized small models trained on step-by-step reasoning to handle high-throughput workloads.

AI-assisted summary of Making Generative Models Work in the Real World: From LLMs to SLMs With Tinder — original source. Check the original for full context.

Source published:

Detailed notes

## Background Published on YouTube by Ai4 on August 27, 2025, this talk features Sanket Badhe, a Staff Machine Learning Engineer at Tinder, addressing the transition from generative prototypes to production systems. According to the page description, the session explores why adapting large foundational models into smaller, faster architectures is essential for interactive, high-traffic environments, detailing practical strategies for model design, training workflows, and deployment trade-offs.

## TL;DR Sanket Badhe outlines why consumer-scale applications must transition away from massive, generalized LLMs toward specialized Small Language Models (SLMs) to satisfy strict 30-to-40 millisecond latency requirements and reduce operational costs. Production LLM pipelines suffer from an expensive 'generality tax,' third-party privacy concerns, and an unpredictable 'prompt hamster wheel' when attempting to patch edge cases. To overcome this, Tinder employs Parameter-Efficient Fine-Tuning (LoRA) and distilled chain-of-thought reasoning over ambiguous 'gray-area' data to train focused, modular models on a single GPU. The presentation concludes by presenting a hybrid architecture where fast, lightweight SLMs handle the vast majority of routine traffic, falling back to frontier LLMs or human moderators only for low-confidence edge cases.

## Atmosphere & style The presentation carries the pragmatic, battle-tested cadence of an experienced systems architect cutting through modern generative AI hype. Badhe moves through a distinct narrative arc that begins with early wonder at zero-shot prompt comprehension, quickly pivoting into the sober, unglamorous reality of SLA bottlenecks, cloud invoices, and fragile regressions. Listening to the talk feels like sitting in on a candid post-mortem and engineering blueprint, marked by grounded metaphors and practical lessons forged directly on the consumer frontlines.

## Key takeaways - [00:04](https://www.youtube.com/watch?v=qTVRPAYFffA&t=4s) Early LLM adoption revolutionized trust and safety workflows by understanding nuanced context, tone, and multi-tiered platform guidelines far better than classical machine learning models. - [02:14](https://www.youtube.com/watch?v=qTVRPAYFffA&t=134s) Scaling commercial LLM APIs to high-volume production introduces severe bottlenecks, including strict 30–40 ms latency requirements, escalating token costs, and data privacy hurdles. - [02:14](https://www.youtube.com/watch?v=qTVRPAYFffA&t=134s) Production LLMs incur a heavy 'generality tax' and create a 'prompt hamster wheel,' where patching edge cases breaks baseline prompts while paying for unnecessary broad reasoning capabilities. - [05:05](https://www.youtube.com/watch?v=qTVRPAYFffA&t=305s) Small Language Models (1B–13B parameters) provide a specialized alternative to generalized LLMs, running up to 100x faster and enabling low-cost deployment on standard CPUs and edge hardware. - [07:22](https://www.youtube.com/watch?v=qTVRPAYFffA&t=442s) Parameter-Efficient Fine-Tuning (PEFT/LoRA) is the most viable strategy for production SLMs, allowing teams to freeze base weights and adapt models for specific domains on a single GPU. - [10:54](https://www.youtube.com/watch?v=qTVRPAYFffA&t=654s) Engineering teams must avoid the 'ROI trap' of self-hosting SLMs by factoring in the hidden operational costs of MLOps, continuous evaluation, and infrastructure maintenance. - [10:54](https://www.youtube.com/watch?v=qTVRPAYFffA&t=654s) Under the production machine learning 'Iron Triangle' (latency, accuracy, and generality), SLMs deliberately sacrifice broad generality to achieve ultra-low latency and high task-specific accuracy. - [12:50](https://www.youtube.com/watch?v=qTVRPAYFffA&t=770s) Effective knowledge distillation transfers the teacher model's step-by-step reasoning logic and policy justifications to the student model rather than merely copying final classification labels. - [12:50](https://www.youtube.com/watch?v=qTVRPAYFffA&t=770s) Dataset quality matters more than raw volume; fine-tuning sets must concentrate heavily on difficult 'gray-area' edge cases and subtle boundary conditions rather than obvious examples. - [15:24](https://www.youtube.com/watch?v=qTVRPAYFffA&t=924s) Production fine-tuning requires a continuous MLOps loop that logs failures, updates adapter weights against regressions, and adapts to evolving adversarial user behavior. - [15:24](https://www.youtube.com/watch?v=qTVRPAYFffA&t=924s) Hybrid production architectures achieve optimal balance by using low-latency SLMs for routine traffic while routing ambiguous or low-confidence edge cases to frontier LLMs or human reviewers.

## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=qTVRPAYFffA&t=4s) Introduction and the Magic of Early LLM Adoption

Sanket Badhe, a Staff Machine Learning Engineer at Tinder with extensive industry background in trust and safety, introduces his talk on bridging the gap between generative AI prototypes and robust, large-scale production systems. While large language models (LLMs) fundamentally revolutionized the artificial intelligence landscape, shipping generative AI into real-world consumer applications requires moving beyond initial proof-of-concept demos and Jupyter notebooks. Rather than merely debating parameter counts, Badhe frames the industry’s evolution as a practical transition toward nimble, specialized Small Language Models (SLMs) capable of serving millions of active users reliably and cost-effectively.

Badhe recalls the early excitement of integrating LLMs into content moderation workflows. For years, trust and safety teams had struggled to categorize user-generated comments and ambiguous text against complex, multi-tiered community guidelines using classical machine learning models. Nuanced, context-heavy edge cases routinely fell through the cracks. However, when Badhe supplied a single structured prompt containing the platform's moderation guidelines to an LLM, the model performed almost miraculously—comprehending tone, contextual nuance, and underlying user intent in a manner prior architectures could never achieve.

[02:14](https://www.youtube.com/watch?v=qTVRPAYFffA&t=134s) The Hidden Costs and the "Generality Tax" of Large Models in Production

The moment teams attempt to scale LLM prototypes to a massive user base, severe operational bottlenecks emerge. In interactive moderation workflows, platform SLAs require sub-second processing—often with strict latency thresholds around 30 to 40 milliseconds—rendering typical cloud API response times of 1 to 2 seconds completely unusable. Compounding the latency problem are ballooning financial costs; API charges that seem negligible on a per-token basis quickly escalate into massive cloud invoices at scale, forcing organizations into painful compromises between user experience and bottom-line budgets. Furthermore, processing sensitive user data through third-party managed APIs introduces critical privacy, data residency, and enterprise compliance hurdles.

```mermaid flowchart TD accTitle: The Generality Tax of LLMs in Production subgraph S1["LLM Capabilities vs Task"] A["Massive LLM Capabilities<br/>(Poetry, Quantum Physics, Coding)"] -->|Applied to| B["Narrow Production Task<br/>(Policy Violation Check)"] end subgraph S2["Production Bottlenecks"] B --> C["Strict Latency Constraints<br/>(30-40 ms required)"] B --> D["Exploding API Bills & Data Privacy"] B --> E["Prompt Hamster Wheel<br/>(Fixing 1 case breaks others)"] B --> F["Generality Tax<br/>(Unpredictable outputs & high compute)"] end ```

Engineers also face what Badhe terms the "prompt hamster wheel," a frustrating cycle where an engineered prompt handles 80% of standard queries, but patch attempts for rare edge cases inadvertently break previously working scenarios. On top of excessive GPU power and energy footprints, deploying massive models incurs a heavy "generality tax." Organizations end up paying for a model capable of writing poetry, solving quantum physics, and drafting legal briefs when all they require is a strict policy violation classifier. In Badhe's words, using an all-purpose LLM for a dedicated moderation task is like "trying to use a sledgehammer to crack a nut."

- **Strict SLA Targets**: Real-time content moderation demands decisions within 30 to 40 ms, whereas commercial LLM APIs often take 1,000 to 2,000 ms. - **The "Prompt Hamster Wheel"**: Iterative prompt tweaks to fix fringe edge cases frequently degrade accuracy across baseline benchmark sets. - **The Generality Tax**: Paying for broad reasoning breadth introduces unnecessary latency, cost, and output non-determinism into narrow production pipelines.

[05:05](https://www.youtube.com/watch?v=qTVRPAYFffA&t=305s) Defining Small Language Models (SLMs) and Performance Advantages

Small Language Models (SLMs) represent a fundamental paradigm shift from "bigger is always better" to selecting the "right tool for a right job." While labeled "small," modern SLMs typically span 1 billion to 13 billion parameters—a footprint that is only small when contrasted with frontier models scaling into hundreds of billions or trillions of parameters. Unlike generalized LLMs, SLMs are purpose-built to act as deep vertical experts rather than generalists, prioritizing minimal compute footprints, low operational overhead, and ultra-fast inference.

| Metric / Dimension | Large Language Models (LLMs) | Small Language Models (SLMs) | |:--- |:--- |:--- | | **Parameter Scale** | $\sim 100\text{B} - 1\text{T}+$ parameters | $\sim 1\text{B} - 13\text{B}$ parameters | | **Inference Latency** | High (Seconds; unsuited for fast SLAs) | Ultra-low (Up to $100\times$ faster; sub-50 ms capable) | | **Operational Cost** | High variable API pricing / heavy multi-GPU clusters | Extremely low; hostable on standard compute/CPUs | | **Deployment Target** | Cloud data centers / API endpoints | Edge devices, CPUs, and local infrastructure | | **Core Philosophy** | Broad generalist (poetry, code, general reasoning) | Highly specialized task expert |

Because SLMs can run up to $100\times$ faster than frontier models, they unlock entirely new real-time product features that were previously non-viable. Their compact footprint enables deployment directly on local CPUs and edge devices, drastically lowering operational carbon footprints while eliminating dependency on external API availability.

[07:22](https://www.youtube.com/watch?v=qTVRPAYFffA&t=442s) Strategies for Building SLMs: Pre-training, Distillation, and Fine-Tuning with LoRA

When implementing an SLM, engineering teams generally evaluate four architectural pathways:

```mermaid flowchart LR accTitle: SLM Implementation Spectrum subgraph Pathways["SLM Implementation Pathways"] direction TB P1["1. Off-the-Shelf SLMs<br/>(Llama 3 8B, Gemma)"] P2["2. Training from Scratch<br/>(Heavy compute; AI labs only)"] P3["3. Knowledge Distillation<br/>(Soft labels & teacher reasoning)"] P4["4. Parameter-Efficient Fine-Tuning<br/>(LoRA on single GPU)"] end P1 -->|Quick start, hits limits| Output["Specialized Production Model"] P3 -->|Transfers teacher logic| Output P4 -->|Preferred: High quality & low cost| Output ```

1. **Pre-trained Open-Source Models**: Leveraging community releases such as Meta's Llama 3 8B or Google's Gemma models out of the box. While prompting off-the-shelf SLMs works for generic tasks, they hit a performance ceiling on nuanced domain edge cases. 2. **Training from Scratch**: Building foundational architectures from the ground up requires terabytes of domain data and massive compute clusters. Badhe notes this approach is impractical for standard enterprise teams and is best left to foundational AI research labs. 3. **Knowledge Distillation**: A large teacher model (e.g., GPT-4) generates probability distributions ("soft labels") across potential outcomes for a student SLM to learn. This enables the smaller model to absorb not just *what* decision to make, but *why*, though it also risks inheriting the teacher's latent biases. 4. **Parameter-Efficient Fine-Tuning (PEFT / LoRA)**: Freezing the core base weights of a pre-trained small model and training lightweight low-rank adapter (LoRA) layers using curated, high-quality domain data. This strategy proved optimal for Tinder, enabling the team to customize models for production on a single GPU while drastically reducing training times.

[10:54](https://www.youtube.com/watch?v=qTVRPAYFffA&t=654s) Production Realities: The ROI Trap and the Iron Triangle

Badhe warns against falling into the "ROI trap." Simple financial calculations often compare third-party LLM API per-token costs against the raw compute bill of self-hosting an SLM, creating the illusion of an instant, massive cost saving. However, teams routinely overlook the substantial hidden overhead of end-to-end MLOps: managing custom training pipelines, building continuous evaluation tooling, provisioning high-availability serving infrastructure, and dedicating engineering resources to scale capacity. By shifting away from managed APIs, organizations assume total operational ownership.

```mermaid flowchart TD accTitle: The Machine Learning Iron Triangle subgraph Triangle["The Production Iron Triangle"] direction TB T1["Ultra-Low Latency"] T2["High Task Accuracy"] T3["Broad Generality"] end T1 --- T2 T2 --- T3 T3 --- T1 subgraph Tradeoff["SLM Trade-off Choice"] Choice["Sacrifice Generality<br/>Focus 100% on Latency + Accuracy"] end Triangle -.-> Choice ```

System architects must also navigate the "iron triangle" of machine learning engineering: balancing ultra-low latency, high accuracy, and broad generality. In production, achieving all three simultaneously is impossible. SLM deployments deliberately sacrifice broad generality to maximize latency and task accuracy, requiring product teams to have unambiguous task specifications before fine-tuning.

[12:50](https://www.youtube.com/watch?v=qTVRPAYFffA&t=770s) Knowledge Distillation Done Right and High-Quality Gray-Area Data

Badhe emphasizes that naive knowledge distillation often produces "a parrot and not a partner." A model that merely mimics the final classification label of a frontier LLM will reliably collapse when confronted with novel, ambiguous distribution shifts. To build a true partner, engineers must distill the complete chain of reasoning: instructing the teacher model to articulate which specific policy was breached, why the breach occurred, and the step-by-step logic behind the verdict before training the student SLM.

Furthermore, increasing dataset size yields diminishing returns if the data lacks ambiguity. Training datasets saturated with obvious examples do not resolve real-world failures. Instead, engineering teams must spend the bulk of their effort curating difficult "gray-area" edge cases—such as nuanced harassment masked as sarcasm—where even professional human moderators might debate the outcome.

- **Reasoning Distillation**: Rather than passing raw binary labels, prompts extract structured justifications explaining why a policy was violated and how the conclusion was reached. - **Prioritizing Gray Areas**: Model accuracy improves substantially when fine-tuning data concentrates heavily on boundary conditions, sarcasm, and subtle subtext over repetitive, unambiguous inputs.

[15:24](https://www.youtube.com/watch?v=qTVRPAYFffA&t=924s) Continuous Fine-Tuning Loops and Hybrid Production Architectures

Fine-tuning is not a static, one-time project; it is an ongoing engineering feedback loop driven by constantly evolving adversarial behavior and shifting user language patterns. The production pipeline systematically logs failures, evaluates model regressions, incorporates human-in-the-loop moderation or LLM labeling for misclassifications, refines the gold-standard dataset, and redeploys updated adapters in rapid iterations.

```mermaid flowchart LR accTitle: Production Feedback Loop and Hybrid Architecture subgraph Loop["Continuous Fine-Tuning Loop"] FT["Fine-Tune SLM"] --> Eval["Evaluate Benchmarks"] Eval --> Fail["Analyze Edge Failures"] Fail --> Relabel["Relabel (LLM / Human)"] Relabel --> FT end subgraph Hybrid["Hybrid Serving Pipeline"] Req["User Request"] --> FastSLM["Specialized SLM<br/>(Fast / Routine Traffic)"] FastSLM -->|High Confidence| Action["Automated Action"] FastSLM -->|Low Confidence / Gray Boundary| Fallback["Multimodal GPT-4 / Human Review"] end ```

Mastering this rapid adaptation loop unlocks modular scaling: rather than relying on a single monolithic model, teams can deploy dozens of distinct, highly focused SLM adapters originating from a shared base architecture. Finally, Badhe highlights the hybrid architecture as the modern production frontier. High-throughput, low-latency SLMs process the vast majority of standard user traffic at minimal cost, while low-confidence boundary edge cases are routed to human moderators or frontier multimodal LLMs like GPT-4 for complex adjudication.

## Notable quotes > [02:14](https://www.youtube.com/watch?v=qTVRPAYFffA&t=134s) trying to use a sledgehammer to crack a nut. — Sanket Badhe

> [02:14](https://www.youtube.com/watch?v=qTVRPAYFffA&t=134s) prompt hamster wheel — Sanket Badhe

> [02:14](https://www.youtube.com/watch?v=qTVRPAYFffA&t=134s) generality tax — Sanket Badhe

> [05:05](https://www.youtube.com/watch?v=qTVRPAYFffA&t=305s) bigger is always better — Sanket Badhe

> [05:05](https://www.youtube.com/watch?v=qTVRPAYFffA&t=305s) right tool for a right job — Sanket Badhe

> [10:54](https://www.youtube.com/watch?v=qTVRPAYFffA&t=654s) ROI trap — Sanket Badhe

> [10:54](https://www.youtube.com/watch?v=qTVRPAYFffA&t=654s) iron triangle — Sanket Badhe

> [12:50](https://www.youtube.com/watch?v=qTVRPAYFffA&t=770s) a parrot and not a partner — Sanket Badhe

## Mentioned Sanket Badhe, Tinder, Ai4, Meta, Llama 3 8B, Google, Gemma, GPT-4, LoRA, PEFT, Jupyter

## Video description Sanket Badhe, Staff Machine Learning Engineer, Tinder

Abstracts

Generative models are transforming how interactive features are built for real-time, high-traffic environments. This talk will illustrate why adapting large models into smaller, faster versions is essential for specific tasks. Attendees will learn practical strategies for model design, training workflows, and effective evaluation and deployment. I will also explain key trade-offs involving latency, quality, and output alignment. Join to discover what really works when bringing generative models into production.

[Source](https://www.youtube.com/watch?v=qTVRPAYFffA)

中文简报

Tinder部署微调小模型大幅降低审核延迟 在生产环境中部署通用LLM会带来无法承受的延迟与成本瓶颈，需要采用经过逐步推理训练的专用小模型来应对高吞吐量工作负载。 Sanket Badhe阐述了为何消费级应用必须从庞大的通用大语言模型（LLM）转向专用的小语言模型（SLM），以满足30至40毫秒的严格延迟要求并削减运营成本。生产环境中的LLM管线承受着高昂的“通用性税”、第三方隐私隐患，以及在尝试修复边缘用例时陷入难以预测的“提示词跑轮陷阱”。为了解决这些问题，Tinder采用参数高效微调（LoRA）并针对模棱两可的“灰色地带”数据应用蒸馏思维链推理，在单张GPU上训练出专注且模块化的模型。演讲最后介绍了一种混合架构：由快速、轻量的SLM处理绝大部分常规流量，仅在低置信度的边缘用例中才回退到前沿LLM或人工审核员。 早期LLM的应用彻底变革了信任与安全工作流程，其对细微语境、语气和多层级平台准则的理解能力远优于传统机器学习模型。 (0:04) 将商业LLM API扩展至大规模生产环境会遭遇严重瓶颈，包括严格的30–40毫秒延迟要求、不断攀升的Token成本以及数据隐私障碍。 (2:14) 生产级LLM会产生沉重的“通用性税”并导致“提示词跑轮陷阱”——修复边缘用例往往会破坏基础提示词，同时还要为不必要的宽泛推理能力付费。 (2:14) 小语言模型（1B–13B参数）提供了替代通用LLM的专用方案，运行速度最高可提升100倍，并支持在标准CPU和边缘硬件上进行低成本部署。 (5:05) 参数高效微调（PEFT/LoRA）是生产级SLM最可行的策略，它允许团队冻结基础权重，仅用单张GPU即可针对特定领域适配模型。 (7:22)

Reading summary

Tinder Staff Machine Learning Engineer Sanket Badhe argues that deploying generative AI at scale requires shifting from monolithic, general-purpose large language models to specialized small language models integrated within hybrid production pipelines. The Production Bottlenecks of Generalist LLMs (2:14) While early proofs-of-concept showed that zero-shot prompts in large language models (LLMs) could parse complex moderation guidelines far better than classical machine learning models, real-world deployment exposed severe operational bottlenecks. Real-time interactive moderation requires platform latencies between 30 and 40 milliseconds, whereas commercial cloud LLM APIs typically require 1,000 to 2,000 milliseconds. Scaling these cloud APIs also triggers escalating token costs and raises data privacy and compliance concerns when transmitting sensitive user data to third-party endpoints. Furthermore, engineering teams encounter what Badhe terms the 'prompt hamster wheel,' where modifying prompts to address rare edge cases inadvertently breaks previously functional baseline behaviors. Deploying massive models incurs a 'generality tax,' requiring organizations to fund massive compute and energy capacity for capabilities like creative writing or coding when the production pipeline only requires a deterministic policy violation classifier. Specializing Small Language Models via LoRA (5:05) Small Language Models (SLMs), typically defined between 1 billion and 13 billion parameters, prioritize minimal compute footprints and fast inference over broad generality. Because SLMs can run up to 100 times faster than frontier models, achieve sub-50 millisecond latencies, and run on local CPUs or edge infrastructure, they make real-time product constraints technically viable while eliminating dependency on external APIs. Among the architectural pathways for implementing SLMs—pre-trained models out of the box, training from scratch, knowledge distillation, and parameter-efficient fine-tuning (PEFT)—Badhe highlights Low-Rank Adaptation (LoRA) as optimal for production teams. By freezing base model weights and training lightweight adapter layers with high-quality domain data on a single GPU, teams can tailor models to specialized operational tasks without the unsustainable compute burdens of training from scratch. Navigating Trade-Offs and Distilling Model Reasoning (10:54) Adopting self-hosted SLMs requires navigating what Badhe calls the 'iron triangle' of machine learning engineering: balancing low latency, high task accuracy, and broad generality. In production, achieving all three simultaneously is impossible; SLMs succeed by deliberately sacrificing generality. Teams must also avoid the 'ROI trap' of comparing third-party API token fees only against raw compute costs, which overlooks the substantial overhead of custom MLOps pipelines, continuous evaluation tooling, and infrastructure management. To ensure distilled SLMs perform reliably, Badhe warns against simply training models on raw binary labels, which creates 'a parrot and not a partner' prone to failure under distribution shifts. Effective distillation instead transfers step-by-step reasoning chains from a teacher model, explaining precisely why a policy was violated. Furthermore, training datasets yield the highest return when concentrated on difficult, ambiguous 'gray-area' edge cases—such as subtle harassment concealed as sarcasm—rather than voluminous, obvious examples. Continuous Adaptation in Hybrid Production Architectures (15:24) Model fine-tuning is an active engineering feedback loop rather than a one-time deployment, as user language and adversarial tactics continuously shift. Production pipelines must systematically log edge failures, evaluate model regressions, relabel ambiguous cases using human reviewers or frontier models, and iteratively deploy updated LoRA adapters on top of a shared base model. Badhe highlights a hybrid serving pipeline as the practical operational frontier for modern platforms. In this setup, specialized, low-latency SLMs process the vast majority of routine user traffic at minimal compute cost, while low-confidence predictions and boundary edge cases are dynamically escalated to human review or frontier multimodal models like GPT-4 for complex adjudication. While specialized small language models resolve the latency, cost, and reliability bottlenecks of pure frontier LLM deployments, organizations must be prepared to absorb the continuous MLOps overhead of data curation, reasoning distillation, and infrastructure maintenance to sustain them in production.

精炼阅读摘要

Tinder主任机器学习工程师Sanket Badhe指出，将生成式AI从概念验证推向高并发生产环境，核心路径是从臃肿昂贵的通用大语言模型（LLM）转向针对特定任务优化的小语言模型（SLM）。在内容审核等实时场景中，团队需主动舍弃不必要的通用能力，以攻克严苛的延迟、成本与隐私瓶颈。 通用LLM的生产瓶颈与通用性税 (2:14) Badhe回顾指出，通用大语言模型在内容审核等信任与安全场景中展现出前所未有的语境与意图理解能力，但当原型尝试扩展至数百万用户时，严重的工程瓶颈随之爆发。交互式审核流水线通常要求30至40毫秒的极低处理延迟，而商业云端API常见1至2秒的响应时间完全无法满足业务SLA。同时，按Token计费的API在海量调用下导致成本剧增，将敏感用户数据传输至第三方服务也带来了数据驻留与合规风险。 工程师在维护通用提示词时还会陷入所谓的“提示词仓鼠轮”困境，即针对长尾边缘案例微调提示词时，往往会反向破坏原有基准场景的表现。Badhe强调，部署庞大模型让企业承担了沉重的“通用性税”：系统不仅要为写诗、推理量子物理或起草法律文书等多余能力支付计算和能耗开销，而实际业务需要的仅仅是一个专注的合规分类器。这种在专用场景调用全能大模型的做法，如同“用大锤砸坚果”。 SLM优势、LoRA微调与ROI陷阱 (5:05) 现代小语言模型（SLM）的参数量通常在10亿至130亿（1B–13B）之间。SLM的推理速度可达前沿大模型的100倍，能够直接部署在常规服务器、CPU乃至边缘设备上，不仅消除了对外部API的可用性依赖，还大幅降低了碳足迹。在对比完全开源自建、从零训练、知识蒸馏以及参数高效微调（PEFT/LoRA）等路径后，Tinder团队认定冻结基座权重并基于领域数据训练LoRA轻量适配层是最优解，仅需单张GPU即可完成生产级定制。 在架构取舍上，工程师必须面对超低延迟、高任务准确度与广泛通用性不可兼得的机器学习“铁三角”。SLM部署策略明确选择牺牲通用性，以此换取极限的延迟表现与高垂直准确度。Badhe同时警告团队防范简单的“ROI陷阱”：单纯对比自建算力与第三方API调用费容易产生暴利节省的错觉，却忽略了构建自定义训练流水线、持续评估工具集、高可用性部署以及工程扩容等全套MLOps基础设施所带来的高昂隐形成本。 推理链蒸馏与灰色地带数据构建 (12:50) 针对知识蒸馏，Badhe强调粗糙的蒸馏往往只会产出“学舌的鹦鹉，而非协作的伙伴”。如果小型模型仅学习前沿大模型的最终分类标签，在遭遇未见过的模糊文本分布漂移时便会迅速失效。构建高韧性SLM的关键在于蒸馏完整推理链：促使教师模型（如GPT-4）输出具体的判定逻辑，包括违反了哪项具体条款、违规的核心理由以及得出结论的逐步推导过程，再用该结构化逻辑指导学生模型学习决策原理。 单纯堆砌训练数据规模的边际收益极低，充斥明显违规案例的语料无法解决真实生产故障。Badhe指出，工程力量应当重度倾斜于挖掘和标注高难度的“灰色地带”边缘样本，例如伪装在讽刺语气之下的隐蔽骚扰等连人工审核员都易产生分歧的临界个案。在微调数据中强化此类微妙语境和潜台词的权重，是显著提升模型抗干扰能力的决定性因素。 持续微调闭环与混合生产架构 (15:24) 模型微调不是一次性任务，而是由不断演变的用户语言风格与对抗性行为驱动的长期工程闭环。在Tinder的生产管线中，系统持续监控并记录线上预测失败案例，评估模型性能回退，依托大模型标注或人工复核对误判个案重新打标，将清洗后的黄金数据追加至数据集中，并快速重新部署更新后的LoRA适配层。这一闭环让团队能够基于共享的通用小底座，并行维护并迭代数十个高度专业化的适配模块。 Badhe最后展示了前沿的混合生产服务架构。面对高并发请求，系统首先由轻量、超低延迟的专用SLM处理绝大部分常规流量，实现极低的推理开销；当SLM对低置信度的模糊边界案例无法确信时，请求会被自动降级分流至人类审核员或GPT-4等前沿多模态大模型进行最终裁决，实现了系统吞吐量与判别精度的平衡。 Badhe总结指出，生成式AI在真实消费级业务中落地的核心原则在于“为特定任务选择恰当工具”，而非盲目追求大参数规模。尽管专注于特定场景的小语言模型能够破解实时延迟与高昂成本的难题，但企业在摆脱外部托管API的同时也必须承担全栈MLOps运营的维护职责；未来高吞吐业务的基准方案，必然是小型专用模型与前沿大模型分层协作的混合架构。

Detailed notes — zh

## Background 该视频由 Ai4 频道于 2025 年 8 月 27 日发布在 YouTube 平台。演讲由 Tinder 的 Staff 机器学习工程师 Sanket Badhe 主讲，围绕如何将生成式大模型（LLM）转化为适用于高并发、实时生产环境的小型语言模型（SLM），分享了模型架构设计、训练工作流、系统评估及延迟与质量权衡等实战策略。 ## TL;DR 演讲深入剖析了在 Tinder 实时内容审核等毫秒级高并发场景下，直接调用大型通用 LLM 所面临的高延迟、成本失控和“泛化税”等工程瓶颈。主讲人提出通过参数高效微调（PEFT/LoRA）与推理链知识蒸馏，利用单卡 GPU 构建 1B 至 13B 参数级别的专用 SLM，以换取极低延迟和高准确率。同时，团队必须警惕自建 SLM 带来的全生命周期 MLOps 隐形成本与“不可能三角”权衡。最终，生产落地的最佳实践是建立持续微调工程闭环，并采用由专用 SLM 承接常规流量、GPT-4 多模态及人工审核兜底长尾边缘案例的混合路由架构。 ## Atmosphere & style 整个分享风格极其务实、干练且充满一线实战气息，摈弃了虚无的 AI 概念炒作，直击线上严苛的生产现实。主讲人以清晰严密的工程逻辑，将最初体验大模型“魔法”的震撼与随后遭遇生产瓶颈的清醒反思娓娓道来。整场演讲呈现出一种硬核架构师拆解复杂系统难题的冷静与专业感，给听众带来极具指导价值的技术落地启示。 ## Key takeaways - [00:04](https://www.youtube.com/watch?v=qTVRPAYFffA&t=4s) AI 行业正经历从追求大而全的庞大 LLM 转向小巧、敏捷的专用小型语言模型（SLM）的范式转移。 - [01:11](https://www.youtube.com/watch?v=qTVRPAYFffA&t=71s) LLM 在概念验证和 Demo 中展现出理解复杂语境的极佳效果，但在高并发生产落地时面临残酷的工程挑战。 - [02:14](https://www.youtube.com/watch?v=qTVRPAYFffA&t=134s) 在 Tinder 实时审核场景下，严格的延迟预算仅为 30-40 毫秒，直接调用秒级响应的商业 LLM 在延迟上不可行。 - [02:14](https://www.youtube.com/watch?v=qTVRPAYFffA&t=134s) 企业在通用任务上使用全能 LLM 会付出不必要的“泛化税”（Generality Tax），带来高昂成本、多余随机性与可预测性下降。 - [05:05](https://www.youtube.com/watch?v=qTVRPAYFffA&t=305s) SLM 的设计哲学是培养垂直领域的“专项专家”而非通才，在显著降低算力成本的同时将推理延迟提升达 100 倍。 - [07:22](https://www.youtube.com/watch?v=qTVRPAYFffA&t=442s) 借助预训练底座结合高质量私有数据，通过 LoRA 等参数高效微调（PEFT）技术，仅需单张 GPU 即可完成定制 SLM 训练。 - [10:54](https://www.youtube.com/watch?v=qTVRPAYFffA&t=654s) 警惕“ROI 陷阱”：自建 SLM 虽节省 API 费用，但团队需全权承担数据流、评测体系与扩缩容等全生命周期 MLOps 隐形成本。 - [11:27](https://www.youtube.com/watch?v=qTVRPAYFffA&t=687s) 系统设计面临超低延迟、高准确率与广泛通用性的“不可能三角”，选择 SLM 本质上是主动放弃通用性以换取极致性能。 - [12:50](https://www.youtube.com/watch?v=qTVRPAYFffA&t=770s) 知识蒸馏必须传递大模型的完整推理链（Reasoning Process）与依据，单纯拟合标签只会得到缺乏边缘泛化能力的“复读机”。 - [12:50](https://www.youtube.com/watch?v=qTVRPAYFffA&t=770s) 数据质量远胜于数据数量，模型针对讽刺、对抗言论等“灰色地带”边缘样本的精细化训练是突破性能瓶颈的关键。 - [15:24](https://www.youtube.com/watch?v=qTVRPAYFffA&t=924s) 微调是一个持续的工程闭环，通过常态化失败归因与重新微调，构建应对动态黑产攻击的多 SLM 专家集群。 - [15:24](https://www.youtube.com/watch?v=qTVRPAYFffA&t=924s) 生产落地的理想架构是“SLM + LLM/人工”混合路由系统：常规高频流量由 SLM 处理，临界疑难案件分流至 GPT-4 或人工兜底。 ## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=qTVRPAYFffA&t=4s) 开场与背景介绍：从 LLM 走向 SLM Sanket Badhe 作为 Tinder 的 Staff 机器学习工程师，首先向在场观众问好，并简要介绍了自己的职业背景。他加入 Tinder 仅数月，此前在业界的绝大多数工作经验都集中在“信任与安全”（Trust and Safety）领域。Sanket 指出，如今许多工程团队都在经历同一段旅程——即探索如何让生成式 AI 模型真正走出实验室与 Demo，在真实世界和高并发生产环境中稳定运行。 当前 AI 行业正在发生一场范式转移：从最初震撼全球、功能极其强大的庞大大型语言模型（LLM），逐渐走向更小巧、敏捷的小型语言模型（SLM）。本次分享的核心并非单纯对比模型参数规模的大小，而是探讨如何把一个刚开始让人惊呼“哇（Wow）”的模型，切实转化为可扩展、可落地的线上生产级架构，并剖析那些只有在系统真正上线后才会显现的“隐形成本”与工程教训。 --- ### [01:11](https://www.youtube.com/watch?v=qTVRPAYFffA&t=71s) LLM 的高光时刻与内容审核中的魔法初体验 LLM 的横空出世彻底重塑了 AI 行业。过去看似不可能完成的任务瞬间变成了现实——几秒钟内提炼超长文档、用日常自然语言直接编写代码。Sanket 回忆起自己在工作中首次使用 LLM 的震撼经历：当时团队需要对用户评论等各种文本进行分类，判断其是否违反特定的平台审核政策。 团队在该问题上已经深耕攻坚了数年，但面对各种细微、模糊的边界案例（edge cases）时，传统模型的表现始终难如人意。然而，当他尝试写了一段简短的 Prompt，把各项审核政策融入其中并输入测试样本后，模型展现出的效果“简直就像魔法一样”。模型第一次真正理解了文本背后微妙的上下文语境、内涵以及说话者的真实意图，这在以往的模型体系中是极难做到的。 --- ### [02:14](https://www.youtube.com/watch?v=qTVRPAYFffA&t=134s) 生产环境的隐形成本与“泛化税” 当团队试图将这份“魔法”推向平台数以千万计的真实用户时，一系列残酷的工程挑战与隐形成本随之暴露： - **极高的延迟（Latency）**：在演示或实验阶段，等待 1 到 2 秒完全可以接受；但在实时生产环境里，1 到 2 秒慢得像“过了一个世纪”。在 Tinder 的实时审核场景下，严格的延迟预算仅为 30 至 40 毫秒，调用庞大 LLM 根本不可行。 - **失控的账单（Cost）**：初期看 API 的单 Token 价格极低，仿佛只有几分钱；但随着高并发流量涌入，云端账单会迅速爆炸式增长，迫使团队在“保障用户体验”与“削减成本”之间做艰难抉择。 - **隐私与合规（Privacy）**：审核业务涉及大量敏感数据，调用第三方商业 API 面临繁重的安全审查与合规链路，极大地拖慢了系统迭代。 - **Prompt 仓鼠轮（Prompt Hamster Wheel）**：开发者精心调优的 Prompt 往往能覆盖 80% 的常见场景，但一旦为了修复某个长尾边界案例而修改 Prompt，原本能正常跑通的其他用例就会随之崩溃，陷入无休止的修补恶性循环。 - **能耗与环保（Energy Use）**：大模型依赖庞大的 GPU 集群与高额电力消耗，带来了不容忽视的算力开销与环境负担。 这些现实痛点让团队开始反思：我们正在用一个能写优美诗歌、解答量子物理难题、起草专业法律合同的通用超级大脑，去解决一个仅仅判断是否违规的专用任务。Sanket 将这种为不必要的多样化能力买单的现象称为**“泛化税”（Generality Tax）**。这种多余的能力不仅带来了极高的成本，还引入了额外的随机性，降低了输出的可预测性。这无异于“用大铁锤去砸一颗小坚果”。 --- ### [05:05](https://www.youtube.com/watch?v=qTVRPAYFffA&t=305s) 重新定义 SLM：从全能通才到专项专家 SLM（Small Language Model）并非粗制滥造的“降智版”LLM。尽管被称为“小”模型，它们通常也拥有数十亿（1B 到 13B）的参数量，在传统意义上绝不算小；但在动辄千亿甚至万亿参数的现代大模型浪潮下，它们显得极为轻巧。 SLM 的设计哲学与 LLM 截然不同：它的目标不是打造一个面面俱到的“通才”，而是培养一个针对特定场景做到极致的“专项专家”。这种设计理念标志着工程策略从盲目信奉“越大越好”转向“在正确的场景使用正确的工具”。 | 评估维度 | 大型语言模型 (LLM) | 小型语言模型 (SLM) | 生产环境影响 | |:--- |:--- |:--- |:--- | | **参数规模** | 数百亿至数万亿参数（高出约 100 倍） | 通常在 1B 至 13B 参数级别 | 基础计算负载与内存占用大幅降低 | | **推理延迟** | 秒级响应（1-2 秒以上） | 比 LLM 快达 100 倍（毫秒级） | 解锁高频、极低延迟容忍度的全新实时产品形态 | | **算力与成本** | 依赖昂贵云端 GPU 集群，API 账单高昂 | 推理成本极低，无需反复纠结 API 调用预算 | 可在全产品线大规模铺开，支持本地/边缘部署 | | **部署环境** | 必须部署在大型多卡 GPU 基础设施 | 可运行在通用 CPU 或端侧设备上 | 解决敏感数据外发问题，兼具低能耗与隐私安全 | --- ### [07:22](https://www.youtube.com/watch?v=qTVRPAYFffA&t=442s) 构建 SLM 的四种技术路径 针对如何构建适合业务的 SLM，Sanket 系统性地梳理了四种常见路径，并分析了各自的适用场景与权衡： ```mermaid flowchart TD accTitle: 构建小型语言模型（SLM）的四种路径 subgraph S1["路径概览"] A["1. 开箱即用预训练模型<br/>(Llama 3 8B / Gemma 等)"] B["2. 从零从头预训练<br/>(Pre-train from scratch)"] C["3. 知识蒸馏<br/>(Knowledge Distillation)"] D["4. 参数高效微调<br/>(PEFT / LoRA)"] end subgraph S2["适用性评估"] E["易上手但复杂长尾案例易遇瓶颈"] F["耗资巨大、需海量算力，仅适合顶尖 AI 实验室"] G["利用教师模型软标签，学习 reasoning 但会继承偏见"] H["冻结底座并微调适配层，单卡 GPU 即可完成定制"] end A --> E B --> F C --> G D --> H ``` - **直接使用开源预训练模型**：如 Meta 的 Llama 3 8B 或 Google 的 Gemma。直接下载部署并编写 Prompt 即可运行。对于通用任务很有效，但面对细分垂直领域的复杂语境时会迅速触碰性能天花板。 - **从零从头预训练（From Scratch）**：需要数 TB 级数据和庞大 GPU 集群，对于绝大多数企业和普通开发者既不现实也无必要，这是 OpenAI 或 Google DeepMind 等顶级 AI 实验室的任务。 - **知识蒸馏（Knowledge Distillation）**：这是一种极其优雅且实用的方案。使用 GPT-4 等超大模型作为“教师模型”执行任务，提取其输出的各种概率分布（软标签，Soft Labels）来训练较小的“学生模型”。学生模型不仅能学到“答案是什么”，还能领悟“为什么这样判断”。但需要注意，大模型自身的偏差与幻觉也会被蒸馏传递。 - **基于高质量数据的微调（Fine-Tuning / PEFT）**：这是 Tinder 团队践行后认为最契合自身业务的路径。借助预训练底座已具备的语言与逻辑理解力，使用团队精选的高质量私有业务数据，利用 LoRA（Low-Rank Adaptation）等参数高效微调技术进行训练。无需改动原本的全部 70 亿参数，只需冻结底座权重并训练轻量的 Adapter 适配层，大幅降低了计算复杂度。在实际落地中，他们仅凭**单张 GPU** 就成功完成了专属 SLM 的训练。 --- ### [10:54](https://www.youtube.com/watch?v=qTVRPAYFffA&t=654s) 警惕“投资回报率陷阱”（ROI Trap） 在开始深入分享部署经验时，Sanket 首先指出了一个极易误导团队的误区——“ROI 陷阱”（ROI Trap）。 从表面上看，商业账本的算术逻辑似乎再简单不过：开发者先算出业务规模下调用商业 LLM API 的总费用，再估算一下自己搭建并托管一个开源 SLM 的纯推理服务器成本。两者对比往往会呈现出极其诱人的断崖式成本节省，使得团队迅速从纯财务角度拍板决定：“自建 SLM 是一笔稳赚不赔的买卖”。然而，真实的落地账本远不止推理硬件这一项。 ### [11:27](https://www.youtube.com/watch?v=qTVRPAYFffA&t=687s) 权衡取舍：隐藏的 MLOps 成本与“不可能三角” 在决定从大型语言模型（LLM）转向小型语言模型（SLM）时，表面上看似乎是一个显而易见且易于做出的选择，但团队往往严重低估了其背后的隐形成本。当调用第三方 LLM API 时，大部分系统复杂性都被外包给了 API 提供商；而一旦选择自建并运行私有 SLM，团队就必须全权负责整个 MLOps 生命周期。这包括端到端的数据处理流水线、模型评估体系、托管与部署架构等基础设施建设。尽管模型体积变小了，工程设计与维护的责任却成倍增加，无论是弹性扩缩容还是日常运维，都需要投入大量工程资源。 ```mermaid flowchart TD accTitle: SLM MLOps 职责与不可兼得的三元悖论 subgraph MLOpsLifecycle ["自建 SLM 的 MLOps 完整生命周期职责"] A["数据处理流水线"] --> B["微调与蒸馏训练"] B --> C["系统化评估工具"] C --> D["模型托管与弹性扩缩容"] end subgraph IronTriangle ["不可能三角 (Iron Triangle) 的权衡"] T1["超低延迟 (Ultra-low Latency)"] T2["高准确率 (High Accuracy)"] T3["广泛通用性 (Broad Generality)"] T1 --- T2 T2 --- T3 T3 --- T1 end ``` 此外，构建模型时必须面对由“超低延迟”、“高准确率”和“广泛通用性”构成的“不可能三角”（Iron Triangle）。在实际业务场景中，三者不可能同时兼得。采用 SLM 本质上是主动放弃了通用性，因此在着手构建应用前，必须设立极其明确的业务目标，针对特定垂直场景进行针对性微调，切忌指望小模型具备通识通用能力。 - **全流程 MLOps 负担**：自研与托管 SLM 需自主承担数据清洗、模型训练、评测套件及在线服务扩缩容等全套工程基础设施。 - **放弃通用性换取极致效能**：SLM 必须专注于特定任务（如内容审核），明确目标边界以换取超低延迟与垂直任务的高精度。 --- ### [12:50](https://www.youtube.com/watch?v=qTVRPAYFffA&t=770s) 知识蒸馏的关键：传递推理链条而非制造“复读机” 从理论上看，知识蒸馏的逻辑非常简单：利用 GPT-4 等顶尖大模型生成输出标签，再用这些数据去训练小模型。然而在工程实践中，“单纯的蒸馏只会给你带来一只鹦鹉，而不是一个合伙人”（"distillation alone gives you a parrot and not a partner"）。鹦鹉可以模仿听到的声音，却无法理解背后的深层含义。单纯拟合最终标签的蒸馏模型，一旦遇到新出现的边缘用例（Edge Cases），表现就会迅速崩溃。 ```mermaid flowchart LR accTitle: 传统蒸馏与全流程推理蒸馏对比 subgraph Traditional ["传统蒸馏 (模仿结果)"] In1["输入文本"] --> LLM1["大模型 (LLM)"] LLM1 --> Out1["最终判定标签"] Out1 --> SLM1["SLM 拟合 -> 易在边缘用例崩溃"] end subgraph ReasoningDistill ["推理过程蒸馏 (学习思考)"] In2["输入文本"] --> LLM2["大模型 (LLM)"] LLM2 --> Reason["违规原因 + 具体条款 + 判定逻辑"] Reason --> SLM2["SLM 深度理解 -> 具备泛化与判别能力"] end ``` 为了让 SLM 具备真正的决策能力，必须对大模型的完整推理链（Reasoning Process）进行蒸馏。以 Tinder 的内容审核场景为例，不仅要让模型判断文本是否违规，还要让其阐述“为什么认为该文本违规”、“具体违反了哪一条社区准则”以及“如何一步步得出该结论”。与此同时，单纯增加训练数据量并不能消除边界失效问题，数据的“质”远重于“量”。必须在训练集中大量引入极具争议的“灰色地带”（Gray Area）样本（例如包裹在讽刺中的仇恨言论等连人类审核员都需激烈讨论的疑难案例），投入精力构建高质量数据集才能根本性提升 SLM 的性能。 - **推理过程蒸馏**：从小模型被动模仿标签，转为系统性学习 LLM 判定过程中的依据、条例与逻辑链路。 - **灰色样本至关重要**：数据盲目扩增无法解决边界失效，需重点挖掘高难度、高争议的边缘案例与反讽言论用于精细化训练。 --- ### [15:24](https://www.youtube.com/watch?v=qTVRPAYFffA&t=924s) 持续工程闭环与“SLM 专家组 + 大模型”混合架构 真实世界中的用户行为、攻击模式及对抗样本（Adversarial Attacks）始终在动态演变，因此模型微调绝不是“一劳永逸”的单次任务，而是一个持续迭代的工程闭环（Continuous Engineering Loop）。团队需要经历“微调模型 ➔ 评估性能 ➔ 归因失败用例 ➔ 引入人工审核或 LLM 重新标注 ➔ 扩充精炼数据集 ➔ 重新微调”的循环。一旦这套敏捷的微调工程闭环跑通，就能解锁全新的架构能力：不再依赖单一 SLM 解决所有问题，而是基于统一的基础模型快速衍生并部署数十个专注于不同细分场景的专用 SLM，避免单个小模型因承担过多任务而出现能力退化。 ```mermaid flowchart TD accTitle: 持续微调工程闭环与混合路由决策系统 subgraph FineTuningLoop ["持续微调工程闭环"] FT["微调 SLM"] --> Eval["性能评估"] Eval --> FailAnalysis["分析失败与边缘案例"] FailAnalysis --> Label["LLM / 人工审核重新标注"] Label --> Refine["提炼扩充高质量数据集"] Refine --> FT end subgraph HybridServing ["混合生产服务架构"] Request["高并发线上请求"] --> Router{"置信度/复杂度分流"} Router -- "常规/高频请求 (低延迟)" --> SLMPool["专用 SLM 专家集群"] Router -- "临界/高难度边缘用例" --> MultiLLM["GPT-4 多模态 / 人工审核"] end ``` 面向生产落地的终极形态则是“SLM + LLM/人工”的混合架构（Hybrid Architecture）。在绝大多数高频、明确的日常请求中，交由专用 SLM 以极低延迟和超高吞吐高效处理；而当置信度落在边界附近的疑难边缘案例上时，系统将其路由至 GPT-4 多模态大模型或人工审核团队进行兜底。这种梯队式协同架构不仅能完美解决边缘用例的准确率短板，还能在成本、延迟与质量之间达成最佳平衡。 - **微调是持续循环**：通过常态化的失效分析与重标注机制，应对黑产与恶意攻击者不断变换的攻击模式。 - **多 SLM 专家矩阵**：单一小模型做多任务会导致退化，应利用基础底模微调出数十个细分领域的专用 SLM 并行运作。 - **混合路由协同**：常规流量由高并发低成本的 SLM 承接，临界疑难案件分流至 GPT-4 多模态或人工审核，实现全链路最佳效能。 ## Notable quotes > [01:11](https://www.youtube.com/watch?v=qTVRPAYFffA&t=71s) 简直就像魔法一样 — Sanket Badhe > [02:14](https://www.youtube.com/watch?v=qTVRPAYFffA&t=134s) 在实时生产环境里，1 到 2 秒慢得像过了一个世纪 — Sanket Badhe > [02:14](https://www.youtube.com/watch?v=qTVRPAYFffA&t=134s) 泛化税（Generality Tax） — Sanket Badhe > [02:14](https://www.youtube.com/watch?v=qTVRPAYFffA&t=134s) 用大铁锤去砸一颗小坚果 — Sanket Badhe > [05:05](https://www.youtube.com/watch?v=qTVRPAYFffA&t=305s) 在正确的场景使用正确的工具 — Sanket Badhe > [10:54](https://www.youtube.com/watch?v=qTVRPAYFffA&t=654s) 自建 SLM 是一笔稳赚不赔的买卖 — Sanket Badhe > [12:50](https://www.youtube.com/watch?v=qTVRPAYFffA&t=770s) distillation alone gives you a parrot and not a partner（单纯的蒸馏只会给你带来一只鹦鹉，而不是一个合伙人） — Sanket Badhe > [12:50](https://www.youtube.com/watch?v=qTVRPAYFffA&t=770s) 数据的质远重于量 — Sanket Badhe > [15:24](https://www.youtube.com/watch?v=qTVRPAYFffA&t=924s) 模型微调绝不是一劳永逸的单次任务，而是一个持续迭代的工程闭环 — Sanket Badhe ## Mentioned Sanket Badhe, Tinder, Meta, Llama 3 8B, Google, Gemma, OpenAI, Google DeepMind, GPT-4, LoRA, PEFT ## Video description Sanket Badhe, Staff Machine Learning Engineer, Tinder Abstracts Generative models are transforming how interactive features are built for real-time, high-traffic environments. This talk will illustrate why adapting large models into smaller, faster versions is essential for specific tasks. Attendees will learn practical strategies for model design, training workflows, and effective evaluation and deployment. I will also explain key trade-offs involving latency, quality, and output alignment. Join to discover what really works when bringing generative models into production. [Source](https://www.youtube.com/watch?v=qTVRPAYFffA)

Source quotes

[1:11] Sanket Badhe: 简直就像魔法一样 [2:14] Sanket Badhe: 在实时生产环境里，1 到 2 秒慢得像过了一个世纪 [2:14] Sanket Badhe: 泛化税（Generality Tax） [2:14] Sanket Badhe: 用大铁锤去砸一颗小坚果 [5:05] Sanket Badhe: 在正确的场景使用正确的工具 [10:54] Sanket Badhe: 自建 SLM 是一笔稳赚不赔的买卖 [12:50] Sanket Badhe: distillation alone gives you a parrot and not a partner（单纯的蒸馏只会给你带来一只鹦鹉，而不是一个合伙人） [12:50] Sanket Badhe: 数据的质远重于量 [15:24] Sanket Badhe: 模型微调绝不是一劳永逸的单次任务，而是一个持续迭代的工程闭环

Source quotes

[2:14] Sanket Badhe: trying to use a sledgehammer to crack a nut. [2:14] Sanket Badhe: prompt hamster wheel [2:14] Sanket Badhe: generality tax [5:05] Sanket Badhe: bigger is always better [5:05] Sanket Badhe: right tool for a right job [10:54] Sanket Badhe: ROI trap [10:54] Sanket Badhe: iron triangle [12:50] Sanket Badhe: a parrot and not a partner

Original transcript

[0:04] Hello uh good morning everyone. Uh I [0:06] hope everyone is uh having a fun at [0:08] conference. Um and I'm really glad to be [0:12] here. Uh my name is Sankit. Um I'm staff [0:14] machine learning engineer at Tinder. Uh [0:17] I joined here a couple of months ago and [0:19] most of my work experience in the [0:21] industry is in trust and safety domain. [0:23] Uh today I want to share about a journey [0:26] that many of us are on. uh making a [0:28] generative model work not in just demo [0:31] not in just u notebook but in the real [0:33] life uh so we look at the shift from [0:36] like a massive powerful large language [0:38] model that we have that change the [0:39] entire AI industry towards more smaller [0:42] agile small language model that are [0:44] changing how are we building like [0:46] products nowadays uh so this is not [0:49] going to be just comparing the model [0:50] sizes but it's more about taking the [0:54] model um from its first Wow impression [0:58] and putting it into production making it [1:00] scalable etc. Uh so I'll also share some [1:03] of the challenges the hidden cost that [1:04] we can only see while uh it's in the [1:06] production and the lessons that I have [1:08] learned from it. [1:11] Uh so let's start with something that we [1:13] all know are great. Uh they changed the [1:16] entire AI industry. Uh suddenly uh the [1:20] things that previously felt impossible [1:22] became real. Like we can summarize the [1:24] long documents in a seconds. you can [1:26] write a code in a plain uh language. Um [1:30] and I still remember the first time I [1:32] used LLMs at my work. Uh so uh we have [1:35] like a different text for many different [1:37] categories such as comment and we tried [1:39] to categorize that into different [1:41] moderation policies and we were working [1:43] on that particular problem for a couple [1:44] of years. Um and still for the nuance [1:48] cases we were not getting the results up [1:49] to the mark. uh but I just created one [1:51] short prompt uh with all our moderation [1:54] policies and just given it some test and [1:57] the results almost feel like magical um [2:00] the moment where model understand the [2:02] nuance the context as well as the intent [2:05] of the u uh the text that was we that [2:09] was really difficult to see pre using [2:11] the previous models. [2:14] Yeah. Uh so but as soon as we take that [2:18] magic and ship it to the millions of [2:21] users that we have that's when we [2:23] encounter like a hidden cost. Um and so [2:27] first is latency. So in demo waiting for [2:30] one or two second is perfectly fine but [2:32] in real time uh but but in the real [2:35] world uh like one or two seconds feels [2:37] like forever and in many uh use cases [2:40] such as moderation our strict latency is [2:42] 30 40 milliseconds. So it's just not [2:44] feasible. Uh the second is cost at at at [2:47] the start. Uh the price per uh token [2:51] APIs looks tiny. It's just like a [2:53] fraction of the cents. But things adds [2:55] up quickly before you know if your uh [2:58] cloud bill uh start exploding and and [3:01] then you need to make a hard choice [3:03] between the customer experience and the [3:05] cost cutting. And the third is the [3:07] privacy. Um so many of us works on the [3:10] sensitive data and uh sending uh this [3:13] data to third party APIs is big deal. We [3:16] need to have like a lot of security [3:17] checks etc. And that's basically uh [3:20] slows things down. And uh the fourth is [3:23] what I call a prompt hamster wheel. So [3:26] basically uh you have written a prompt [3:28] it works for 80% of the cases but as [3:32] soon as um u it start failing for some [3:35] of the age cases as soon as you start to [3:37] fix that for that particular age case it [3:40] start failing for the another cases that [3:42] previously it's working for um and and [3:45] finally energy use so these large [3:47] language models take like massive GPUs [3:50] and uh as you all know it takes a lot of [3:53] electricity to run those and it's [3:55] basically environment mental concern. So [3:57] these challenges basically made us to [4:00] think about our decision. So we are [4:02] using this super smart LLM to solve our [4:05] particular problem. These LLMs can write [4:08] poetry. They can answer like a quantum [4:10] physics uh problem. They can draft legal [4:14] documents etc. But we are just asking it [4:17] to solve a policy viate related [4:19] solution. So then we ask ourselves a [4:22] questions why we are using a model that [4:24] can write a poetry just to check for a [4:26] specific policy violation and uh we [4:29] realize uh what we what I call is like a [4:32] general generality tax. So these models [4:36] has huge range of skills and many times [4:38] it's not useful for us for the task that [4:41] we are trying to do and it make things [4:44] worse. it at randomness it made uh [4:46] results less predictable and and it's [4:49] basically also a big cost to us. [4:52] So um so basically we are trying to use [4:56] like a sledgehammer to crack a nut. Uh [4:59] it is not perfect tool for our uh [5:02] problem right and and that bring us to u [5:05] the SLMs. So what are SLMs really are? [5:09] So these are the image that I have [5:10] generated using AI um and it basically [5:13] captured the spirit of the term. So [5:16] though it call as SLMs uh they have like [5:19] a billions of parameter and and it by no [5:22] any means like a small models but in the [5:24] new landscape where we have like a [5:25] hundreds of billions of parameter or [5:27] trillions trillions of parameter in the [5:29] newer model even the uh like 1 to 13 [5:32] billion models are like a small language [5:35] models but they are actually um I will [5:38] say pretty good models. Uh so what are [5:41] SLMs? SLMs are not like a just dumber [5:44] versions of the uh LLMs. It's basically [5:47] they are models designed for a [5:49] particular task in the mind. They have [5:51] like a different design philosophy. Uh [5:54] the goal is to not create like a perfect [5:56] generalist but to create a perfect [5:58] expert that can solve our particular [6:00] problem. Uh so and they are basically [6:03] optimized for lower latency, lower [6:04] computational cost etc. Uh so this is [6:07] like a strategy shift from bigger is [6:09] always better to right tool for a right [6:12] job. [6:15] Uh so let's talk about subconrete [6:17] numbers. Um uh so the first is [6:21] parameters in the uh LLMs are like 100 [6:25] times more than the SLMs and and this [6:28] particular fact change the all the other [6:29] metrics that you have. Uh so first is [6:32] latency. uh SLMs can be like 100 times [6:35] faster and and this particular 100 time [6:38] faster change the entire uh paradigm [6:41] around how we deploy LLM. So it unlocks [6:45] entirely new products that is not at all [6:47] possible with the SLMs and cost again [6:50] it's a big difference it's um the and [6:53] the basically reduction in the cost [6:55] means you can use SLN throughout your uh [6:59] uh throughout your product and you don't [7:01] need to think more about the uh hosting [7:03] or to uh you don't need to think more [7:07] about calling APIs and the finally is [7:10] energy use so since these models can be [7:13] deployed on the CPU, you can use it for [7:15] the uh on the devices on the for the age [7:19] applications etc. [7:22] Uh yeah, so I will shortly discuss about [7:25] how can we build the SLMs. Uh the most [7:28] simplest way to is to take like a [7:30] pre-trained SLM out of the box. Uh so [7:33] the open source community has done a [7:35] fantastic job for us. uh such as MAS [7:37] Lama 38 billion or Google's GMI model or [7:41] you can just download this model host it [7:43] yourself and just start prompting those [7:45] model as you do with the LLM. Uh but if [7:49] your case is generic that this might [7:51] work but you might quickly hit a ceiling [7:55] um if your case is nuance and it might [7:57] fail for the age cases. And another [8:00] drawback is uh the bas uh I I can cover [8:04] that later actually. Uh the second is [8:07] training a model from scratch. So it is [8:09] exact opposite of the spectrum that we [8:12] just discussed about like using a [8:14] pre-trained model. Uh so I just want to [8:16] be clear it's not feasible to train this [8:18] model from scratch for most of us in [8:20] this particular room because it requires [8:21] like a terabytes of data. Uh it requires [8:25] a lot of GPUs and it's also not needed. [8:27] Um this is basically a task of the large [8:31] um AI labs such as Google deep mind or [8:34] open AAI. Uh the another option is the [8:37] knowledge distillation. So this is this [8:40] like knowledge distillation and the [8:41] finetuning is the most practical way to [8:43] build uh SLM application uh for your [8:46] particular task. So in uh so knowledge [8:49] distillation concept uh here is pretty [8:51] elegant. So what you do you take a big [8:53] large language model such as GPD4 and uh [8:56] you ask it to do a particular task and [8:59] you don't just take its output but you [9:02] consider all the different outcomes with [9:04] the different probability distribution [9:05] and you train a smaller model. Uh we [9:08] call this like different probability [9:09] distribution as soft level and you train [9:11] a student model which is a smaller model [9:13] using that particular probabilities. Uh [9:16] so it just don't learn like uh what but [9:19] it also learn why. uh uh but one of the [9:23] thing is it gets its own biases from the [9:26] large language models that you are using [9:27] and you might need to consider that uh [9:29] and the final and uh I think for us uh [9:33] that uh works best is the fine-tuning [9:35] your own model. So in this particular [9:37] case you start with using a large [9:40] language model um that has all the [9:43] capabilities such as grammar, language, [9:45] reasoning etc. and and then you train uh [9:49] that particular model using your own [9:51] data that you have cur curated a high [9:53] quality data uh and and another good [9:57] thing about is is like you can use a [9:59] technique such as like a parame [10:01] parameter efficient fine-tuning [10:03] uh especially Laura where you don't need [10:06] to train the entire 7 billion parameters [10:08] that this particular model has you can [10:11] freeze the all the different u all the [10:15] different weights the your original [10:17] small language model has and basically [10:19] just train a smaller adapter um um layer [10:22] on top of it and and then basically [10:25] because of that um it's a smaller uh [10:28] layer and because of that the overall [10:29] time complexity decrease uh lot and it [10:32] it requires a reduce like the number of [10:35] GPU uh uh needed in fact you can train a [10:38] small language models using just a [10:40] single GPU that's how we did um so [10:43] basically we get the best of the both [10:45] word we get like our own small language [10:48] model customized for our uh own use [10:51] case. [10:54] Uh yeah I will discuss uh shortly about [10:59] um uh the learnings that I got from uh [11:02] deploying this particular model. The [11:03] first is the ROI trap. Uh so on the [11:07] surface maths look uh incredibly simple [11:09] right? So you calculate the API cost for [11:13] your particular application and you in [11:16] uh you also calculate the inference cost [11:18] if you host your own SLM and um and you [11:22] see like a massive cost saving and and [11:24] and the uh decision is like financial [11:27] no-brainer. You can easily make a [11:28] decision. Uh but we highly underestimate [11:31] the hidden expenses. uh since we are [11:34] basically uh using this model on our own [11:38] uh we need to take care of entire MLOps [11:41] life cycle for this particular model. We [11:43] need to take care of their data [11:44] processing pipeline. We need to take [11:45] care of its evaluation tool. We need to [11:48] take care of hosting setup and etc. Uh [11:50] so when you basically use the uh au LLM [11:56] APIs from the third party source you [11:58] outsource a lot of those complexity to [12:01] the uh the uh API provider right but [12:04] right now you are the owner of all this [12:06] complexity. Uh so though the model [12:09] itself is smaller but it has like a [12:11] bigger design responsibilities if you [12:13] need to scale up scale down you need to [12:15] have all those engineering resources. Uh [12:17] the second is about uh the iron uh [12:21] triangle. So you cannot have low like uh [12:24] all three like ultra low latency, high [12:26] accuracy and a broad uh generality. So [12:29] here you are basically compromising on [12:32] generality and when you are whenever you [12:34] are building your application you need [12:36] to think about it that your model will [12:38] not have actual generality. you need to [12:40] have a super clear goal why you're using [12:44] SLMs and then you need to fine-tune just [12:46] for that particular application. [12:50] Uh second is about uh teaching an SLM to [12:54] decide well uh so in the theory [12:57] knowledge distillation sounds really [12:58] simpler. you take a bigger model such as [13:00] GPD4, you take its output, train the [13:03] smaller language model using those [13:05] output um and and it should work as [13:09] great. But uh I'll say distillation [13:12] alone gives us gives you a parrot and [13:14] not a partner. So what that I mean a [13:17] parrot can mimic whatever it hurts but [13:19] it don't comprehend what's the meaning [13:22] behind it. Similarly, your distill model [13:24] will try to replicate the output of your [13:26] large language model, but it but when it [13:29] sees the newer age cases, it will not [13:31] able to uh it it will probably fails on [13:35] on those particular cases. Um so what [13:38] what we need we need a model to learn [13:40] the reasoning behind uh the uh reasoning [13:44] behind decision. So we just don't need [13:47] to distill the final outcome of your [13:49] large language model but you also need [13:51] to um distill the entire reasoning [13:53] process. So how can we do basically for [13:56] our moderation case we can we answer so [13:58] this is the text why do you think uh [14:00] this particular text like uh is is this [14:03] particular text violating our policy if [14:05] it is uh which particular policy is [14:07] violating and how you decided that. So [14:10] now instead of just uh distilling your [14:13] model on just on the final output you [14:15] are trying to distill uh your model on [14:18] entire reasoning process [14:21] uh and the uh the second and third is [14:25] kind of related. So what we have also [14:26] found we first tried to create like a [14:29] lot of data we already had a lot of data [14:31] we tried to feed it to the um uh [14:35] we tried to feed it to the SLMs and uh [14:38] we try to train it um and what we found [14:42] it started failing on the age cases and [14:44] whatever however we increase the data [14:47] size it again start failing so it not [14:49] just important of the how much data that [14:52] you are feeding to the SLM but also the [14:54] what kind of data [14:55] that you are feeding to the SLMs is also [14:57] really important. What that I mean we [14:59] just we need to feed a lot of gray area [15:03] kind of cases. The gray area cases means [15:05] a tough cases where even the moderate [15:08] moderator have a debate about what kind [15:10] of is it like a policy organization or [15:12] not such as uh hate speech but kind of [15:15] in the sarcasm or so. So we you need to [15:18] create a high quality data set and spend [15:20] a significant amount of time if you want [15:22] to improve the performance of your SLM. [15:24] Yeah. So this is the last slide. So the [15:27] word is constantly changing. The user [15:29] behavior uh constantly evolve. Uh the [15:32] bad actors tries to find a new uh attack [15:35] pattern. Uh the adversarial attacks. Uh [15:39] so what we have learned the model uh so [15:41] basically you just can't have [15:43] fine-tuning at one time and that's it. [15:46] Uh so finetuning is not like a one-time [15:48] event. It's like continuous engineering [15:50] loop. You basically fine-tune a model, [15:53] analyze uh it analyze u basically fine [15:57] tune a model, evaluate its performance, [16:00] analyze the failures, see why it's [16:02] exactly failing, what are those cases, [16:04] what should be the outcome for it, uh [16:06] maybe use human moderator if needed or [16:09] maybe use like a large language model [16:11] again to label those data set, refine [16:13] those data set and repeat. So uh the [16:16] good thing about is that once we have [16:18] this really good efficient engineering [16:21] loop of fine-tuning uh your model really [16:24] quickly uh in a loop uh it it basically [16:28] unlocks the entire new capability. So uh [16:31] you will not just need a one particular [16:33] SLM to solve particular problem. Right [16:35] now you can easily basically scale your [16:38] model and u uh you can deploy like a [16:42] dozens of SLM for individual particular [16:44] cases and you can really easily track [16:46] those cases because remember SLM is an [16:50] expert. If you try to do a lot of things [16:52] using just SLMs its performance kind of [16:54] degrade. So you kind of have one simpler [16:56] base model and try to um uh try to uh [17:01] fine-tune a different sets of model [17:03] based on the different um uh use cases [17:05] that you have. And finally um I will say [17:09] the new frontier is kind of hybrid. You [17:11] can use SLM for most of the use cases [17:13] that you that you have if uh many of the [17:17] uh objective or if it is repeated case. [17:20] Uh but it will fail because it's a [17:22] smaller model. it will fail on the age [17:24] cases and for those age cases you might [17:27] need to have a hybrid case. So if in our [17:30] case if the race uh if the case is [17:32] really on the boundary we send it to [17:34] human moderators and what we have found [17:35] instead of use sending those to the [17:37] human moderator if we can use a GPD4 [17:40] multimodel it can really solve us [17:41] problem. Uh so yeah that's about it. [17:44] Thanks so much. [17:46] [Applause]