Mozilla and CSU Chiefs Map Enterprise AI Customization Cost Traps
Ai4
Fine-tuning generative AI pays off only for edge-case accuracy and model distillation, while long-term deployment costs and vendor feature overlap make full custom builds prohibitive for most enterprise applications.
## Background
This panel discussion was published on YouTube by Ai4 on August 21, 2026. Featuring technology leaders from Vanguard, California State University, Lightning AI, and Mozilla, the session explores enterprise generative AI customization strategies ranging from prompt engineering and RAG to full-scale fine-tuning and domain adaptation.
## TL;DR
Panelists examine how organizations balance performance, cost, and complexity when tailoring frontier and open-weight AI models to specific industry needs. Through concrete examples like healthcare reasoning models, Firefox's local smart tab grouping, and higher education open textbook initiatives, the speakers outline practical decision frameworks for model selection, synthetic data generation, and inference optimization. They emphasize prioritizing evaluation dataset curation over premature fine-tuning and warn against duplicating features already handled by commercial software vendors.
## Atmosphere & style
The session carries a pragmatic, expert-driven tone, structured around insightful moderating from Vanguard's James Belasco and candid contributions from the panelists. The discussion feels grounded and real-world, avoiding hype to focus on practical execution risks, hardware limits, and total cost of ownership. Viewers experience a collaborative exchange between academic and enterprise practitioners sharing both successful deployments and costly missteps.
## Key takeaways
- [00:04](https://www.youtube.com/watch?v=TH6kW8KCDeQ&t=4s) Enterprise customization of generative AI should balance student outcomes, privacy, and cost across a spectrum ranging from prompting and RAG to full fine-tuning.
- [04:13](https://www.youtube.com/watch?v=TH6kW8KCDeQ&t=253s) For administrative operations, organizations should minimize customization to reduce cost, risk, and compliance friction, whereas academic and research settings routinely require fine-tuning open-weight models.
- [04:13](https://www.youtube.com/watch?v=TH6kW8KCDeQ&t=253s) Off-the-shelf models paired with zero/one-shot prompting and RAG solve most enterprise needs; fine-tuning should be reserved for the final 5–10% performance gain or unit-cost optimization.
- [08:42](https://www.youtube.com/watch?v=TH6kW8KCDeQ&t=522s) Fine-tuning a small open model (like T5-Flan to under 60 MB) using synthetic data from larger frontier models enables fast, privacy-preserving, on-device AI features like Firefox's Smart Tab Grouping.
- [14:47](https://www.youtube.com/watch?v=TH6kW8KCDeQ&t=887s) Attempting to custom-build complex AI tools in-house can lead to cost explosions and project cancellations, especially if commercial software vendors are likely to ship similar features off-the-shelf.
- [18:33](https://www.youtube.com/watch?v=TH6kW8KCDeQ&t=1113s) Model serving at scale—not the initial fine-tuning compute—is the primary ongoing cost driver when deploying custom AI solutions.
- [21:02](https://www.youtube.com/watch?v=TH6kW8KCDeQ&t=1262s) Using synthetic data generated by frontier models and validated by human evaluation bypasses data scarcity and protects user privacy during model fine-tuning.
- [24:03](https://www.youtube.com/watch?v=TH6kW8KCDeQ&t=1443s) Multi-layer inference optimization techniques, such as speculative decoding, prompt caching, and model distillation, significantly reduce runtime compute overhead.
- [27:06](https://www.youtube.com/watch?v=TH6kW8KCDeQ&t=1626s) Continuously updating smaller base models with existing fine-tuning datasets and leveraging high-performance engines like vLLM or SGLang lowers the long-term total cost of ownership (TCO).
- [34:08](https://www.youtube.com/watch?v=TH6kW8KCDeQ&t=2048s) To democratize AI effectively, enterprises should focus upskilling efforts on teaching domain experts how to curate high-quality evaluation datasets rather than training deep learning researchers.
- [36:11](https://www.youtube.com/watch?v=TH6kW8KCDeQ&t=2171s) Organizations need centralized AI leadership and clear governance to manage security risks like the 'lethal trifecta' (untrusted content, prompt injection, and data exfiltration).
- [40:27](https://www.youtube.com/watch?v=TH6kW8KCDeQ&t=2427s) Distilling capabilities from a large model (e.g., 120B parameters) into a smaller one (e.g., 4B parameters) serves as a primary strategy for reducing operational inference costs at scale.
The panel kicks off with moderator James Belasco, Head of Data Science for Advice and Wealth Management at Vanguard, introducing the session and inviting each speaker to explain their organization’s strategic mandate for customizing generative AI models.
Ed Clark, CIO for the California State University System (CSU), highlights the massive footprint of CSU as the largest university system in the United States, producing 1 in every 20 college graduates nationwide. For CSU, customization is driven directly by student outcomes: ensuring students graduate on time, stay on track, and secure employment.
Shruti Kamath, Senior Director of AI and Machine Learning at Mozilla (focusing on Firefox), shares her decade-long background in AI, including serving on Amazon's first deep learning team and holding engineering leadership roles at Zillow, PayPal, and Chewy. She outlines Mozilla’s three core pillars for AI customization:
* **Trustworthiness, Transparency & Privacy:** Focusing on local, on-device AI models so user data remains on the device without security or privacy risks over the network.
* **User Agency & Model Choice:** Avoiding single-ecosystem lock-in by letting users bring their own models or mix open-source models within the Firefox browser.
* **Cost vs. Quality:** Balancing the performance trade-offs inherent in model deployment.
Karan Kothari, Principal Engineer at Lightning AI, describes Lightning AI as a cloud provider powering frontier model pre-training for companies like Cursor and Reflection. They also enable enterprises to fine-tune and deploy machine learning workloads, offering tools ranging from Parameter-Efficient Fine-Tuning (PEFT) to reinforcement learning (RL) environments.
---
[04:13](https://www.youtube.com/watch?v=TH6kW8KCDeQ&t=253s) Evaluation Frameworks for Model Customization
Moderator James Belasco transitions the discussion toward decision frameworks: how should an enterprise decide when fine-tuning is necessary versus lighter customization approaches?
Ed Clark outlines a bifurcated framework split between administrative and academic operations:
* **Administrative Strategy:** Adhere to a strict rule of doing the minimum customization necessary to achieve the business objective. Organizations should evaluate cost, risk, and compliance before committing to deep fine-tuning.
* **Academic Strategy:** Leveraging resources like the National Research Platform and CENIC (the shared California AI resource across UC and CSU systems), faculty and students work directly with open-weight models (such as Kimi and DeepSeek) where fine-tuning is routinely required to produce specialized research outputs.
Shruti Kamath emphasizes starting with off-the-shelf models using zero-shot or one-shot prompting, which covers a significant majority of needs. When dealing with proprietary internal data, Retrieval-Augmented Generation (RAG) serves as the logical next step. Fine-tuning should be reserved for squeezing out the final 5% to 10% of performance or optimizing cost and quality trade-offs. She notes that customization spans multiple layers, extending all the way down to the hardware and kernel levels.
- **Prompting & RAG First:** Off-the-shelf models paired with simple prompting or RAG handle general knowledge and proprietary internal retrieval effectively without early fine-tuning.
- **Fine-Tuning Role:** Best applied as an optimization step for edge-case accuracy (the last 5–10%) or cost/speed efficiency.
---
[08:42](https://www.youtube.com/watch?v=TH6kW8KCDeQ&t=522s) Enterprise Use Cases: Healthcare Reasoning & On-Device Tab Grouping
The panelists walk through concrete examples where fine-tuning proved essential to their strategic goals.
Karan Kothari highlights a healthcare client sitting on petabytes of proprietary clinical reasoning data collected from doctors worldwide. Standard frontier or open-weight models performed poorly on specialized medical reasoning out of the box. By fine-tuning the model to reason specifically over clinical evidence retrieved via RAG, they built a highly accurate "doctor's companion" AI that physicians use as a reliable second opinion.
Shruti Kamath presents Firefox’s **Smart Tab Grouping** feature, which automatically groups open tabs and suggests meaningful context-aware names (e.g., recognizing a specific trip being planned with a friend rather than outputting a generic label like "Travel"). To make this work seamlessly across millions of user devices with varying CPU/GPU and memory limits:
* They selected a small base model, **T5-Flan**, fine-tuning it down to a footprint of **under 60 MB**.
* They utilized larger, more powerful models on the backend strictly for synthetic data labeling to generate training data for the smaller model.
* The result doubled overall performance and quality compared to off-the-shelf alternatives while maintaining high speed on local devices.
---
[14:47](https://www.youtube.com/watch?v=TH6kW8KCDeQ&t=887s) Lessons Learned: The Open Textbook Project vs. Smart Manufacturing
Ed Clark shares two candid real-world stories from the California State University system illustrating the financial risks and practical wins of custom AI implementations.
To save students hundreds of millions of dollars in costly course materials, CSU attempted to build an open textbook AI tutor capable of providing active learning and "cognitive dissonance" rather than simply spitting out direct answers. However, restricting the model to a single open textbook proved insufficient for deep learning outcomes. As they expanded the scope to ingest 200+ textbooks and added complex pedagogical guardrails, customization costs exploded. Without a multi-university consortium to share the financial burden, the project was ultimately shelved—underscoring the lesson not to custom-build solutions that commercial vendors are likely to release off-the-shelf.
On the success side, Ed describes a faculty research initiative in smart manufacturing utilizing **YOLO (You Only Look Once)**, an open-source computer vision model. Because off-the-shelf vision models cannot recognize specific factory floor layouts out of the box, fine-tuning on custom imagery was essential to adapt the model to the exact environment and format research outputs accurately.
- **Open Textbook Tutor Failure:** Cost escalated drastically due to dataset expansion (200+ books) and custom pedagogical logic, proving cost-prohibitive for a single university system.
- **YOLO Smart Manufacturing Success:** Custom fine-tuning of image recognition allowed precise object detection tailored to specific factory floor environments.
<ElicitationsGroup message="Explore further details from this panel discussion:">
<Elicitation label="Summarize Part 2 of this panel discussion" query="Provide a detailed walkthrough of Part 2 of the Ai4 panel 'Exploring Fine-Tuning and Customization Strategy Through Effective Use Cases'."/>
<Elicitation label="Compare RAG vs Fine-tuning frameworks" query="Based on the panel discussion, what are the key differences and trade-offs between using RAG vs fine-tuning for enterprise use cases?"/>
</ElicitationsGroup>
[18:33](https://www.youtube.com/watch?v=TH6kW8KCDeQ&t=1113s) Infrastructure Constraints, Base Model Selection, and the Real Costs of Fine-Tuning
The discussion kicks off with a look at the financial realities of model customization. While fine-tuning is often perceived as a compute-heavy and expensive endeavor, Karan Kothari clarifies that the actual fine-tuning process is only a fraction of the total expense. The true primary driver of cost is model serving, particularly when deploying models at a large scale where runtime infrastructure costs compound rapidly.
To navigate this, Karan outlines a decision framework built on three primary "knobs": desired quality, allowable cost, and the required degree of control over both fine-tuning and model serving. Rather than attempting to fine-tune every available base model candidate, organizations must filter their options early by evaluating their serving platform's constraints—such as supported architectures, throughput targets, and latency SLAs.
Furthermore, establishing a rigorous, stable evaluation dataset early in the pipeline is crucial. Without a fixed "last mile" evaluation set, teams end up repeatedly paying for fine-tuning cycles every time their evaluation benchmarks change.
- **Primary Cost Center:** Model serving at scale, rather than the initial fine-tuning compute.
- **The Three Optimization Knobs:** Quality thresholds, budget limits, and system control.
- **Selection Criteria for Base Models:** Aligning candidate models with serving platform constraints, architectural support, and strict latency/throughput SLAs.
- **Cost Reduction Strategy:** Lock down both the "first mile" (data prep) and "last mile" (evaluation metrics) as early as possible to prevent redundant retraining.
---
[21:02](https://www.youtube.com/watch?v=TH6kW8KCDeQ&t=1262s) Synthetic Data, Human Evaluation, and Unifying Institutional Data
Addressing the data requirements for fine-tuning, the panelists detail strategies for overcoming data scarcity without sacrificing quality or privacy. While proprietary data and clear business outcome metrics are ideal starting points, generating sufficient real-world samples is often tedious and time-consuming. Karan explains how teams leverage frontier models to synthesize datasets from a few high-quality, hand-curated examples, shifting the engineering challenge from raw data collection to validating synthetic data fidelity.
Shruti Kamath emphasizes how Mozilla relies heavily on synthetic data generation and data augmentation to respect user privacy. By pairing frontier models for label generation with human evaluations, Mozilla scales training data to the volumes needed for domain adaptation—a methodology they used to train the model behind Firefox's smart tab grouping feature.
From an institutional perspective, Edmund Clark shares how bold, unified goals from a new chancellor forced 22 distinct university campuses within the California State University system to break down operational silos. To prepare for an AI-enabled future, CSU established a unified data platform and a standardized technology "cookbook," creating a shared foundational data layer across all campuses.
> "Instead of the burden becomes how can you sort of evaluate that synthetic data to make sure it really represents your business use case... that is a simpler way to go rather than trying to generate the eval dataset from ground." — Karan Kothari
- **Mozilla's Smart Tab Grouping:** Built using synthetic data validated through human labelers and data augmentation.
- **CSU System Unification:** 22 universities, 22 presidents, and 22 CIOs unified under a single data platform to support system-wide AI initiatives.
- **Privacy-Preserving Workflows:** Synthetic data generation allows organizations like Mozilla to build specialized models without consuming private user data.
---
[24:03](https://www.youtube.com/watch?v=TH6kW8KCDeQ&t=1443s) Multi-Layered Inference Optimization and Enterprise Cost Management
Shruti breaks down the technical stack for controlling inference-time costs into three distinct layers:
1. **Model Layer:** Applying quantization and model distillation to shrink model footprint.
2. **Serving Layer:** Utilizing prompt caching and speculative decoding (pairing a small draft model with a larger target model to speed up generation).
3. **Kernel & Hardware Layer:** Implementing customized GPU kernels tailored to specific workloads and selecting cost-optimal hardware configurations.
Karan cautions that heavy quantization requires careful base model selection; if a domain requires zero quality loss, aggressive quantization may not be viable, reducing down-the-line optimization flexibility.
Edmund adds a real-world perspective on institutional cost shocks. CSU initially leveraged a system-wide ChatGPT agreement with an unstructured "go do whatever you want" mindset. However, vendor policy shifts—such as OpenAI transitioning to credit-based billing—have forced a rapid shift toward tracking usage and imposing limits. Meanwhile, CSU researchers utilize the National Science Foundation-funded National Research Platform, granting access to a 1,000-GPU California setup for faculty research without immediate direct billing.
- **Speculative Decoding:** Uses a small model to draft tokens quickly, passing them to a larger model for verification, reducing expensive compute passes.
- **Quantization Trade-offs:** Precision loss must be tested against quality limits before committing to quantized inference.
- **CSU GPU Infrastructure:** Access to 1,000 GPUs via the NSF-funded National Research Platform for academic research.
---
[27:06](https://www.youtube.com/watch?v=TH6kW8KCDeQ&t=1626s) Total Cost of Ownership: Continuous Fine-Tuning vs. Vendor Lock-In
The conversation turns to Total Cost of Ownership (TCO) and how rapidly shifting base models alter long-term maintenance strategies. Karan argues that fine-tuning should not be a static, one-time event. Because smaller open-weight base models with near-frontier performance are released every few months, re-running existing fine-tuning datasets on newer, smaller base models significantly lowers long-term inference costs. Furthermore, modern serving frameworks like vLLM and SGLang feature native engine optimizations that dramatically boost throughput out of the box.
Shruti outlines Mozilla AI’s approach to managing TCO for new products, such as their in-browser AI "Smart Window." When testing unproven products without clear product-market fit, Mozilla consciously avoids long-term GPU commitments (such as 3- to 5-year reserved contracts) in favor of pay-per-use vendor models. Paying a slightly higher short-term unit price preserves strategic optionality, avoids vendor lock-in, and allows the team to accurately project future infrastructure, maintenance, and human capital costs before making heavy capital investments.
> "If you just change the base model and go through the fine-tuning process again, you can significantly reduce your downstream cost of inference quite a lot." — Karan Kothari
- **Open-Source Engine Optimizations:** Utilizing high-performance inference frameworks like vLLM and SGLang to maximize token throughput.
- **Flexible Infrastructure Strategy:** Choosing on-demand, pay-per-use compute over 3-5 year GPU contracts during early product-market fit discovery.
- **TCO Components:** Infrastructure costs, model maintenance, data updates, continuous evaluation, and dedicated engineering headcount.
---
[32:03](https://www.youtube.com/watch?v=TH6kW8KCDeQ&t=1923s) Replicating Vendor Features vs. Solving Custom Problems
Reflecting on the challenges of product life cycles and institutional execution, Edmund details CSU's experience with a now-paused student companion app. Major textbook publishers (such as Houghton Mifflin Harcourt and Macmillan) are embedding paid AI tutoring components directly into digital courseware. While CSU envisioned providing free, AI-tutored digital materials to eliminate the $400-per-book burden on students, faculty required fully polished, classroom-ready tools immediately—a gap CSU's pilot timing could not bridge.
Edmund warns enterprise teams against duplicating capabilities that commercial software vendors are better equipped to build and maintain. For example, while many organizations consider building custom AI tools to digitize paper transcripts and automate grade entry, document management vendors are already integrating these features directly into off-the-shelf software. Organizations should only build custom fine-tuned solutions when they unlock massive direct savings or solve a uniquely proprietary problem.
- **Publisher AI Integration:** Commercial publishers bundling AI tutors into digital textbooks, competing with internally developed university tools.
- **Textbook Cost Drivers:** Individual business school textbooks costing students up to $400 each.
- **Build vs. Buy Warning:** Avoid spending internal development resources replicating features (like OCR transcript processing) that standard document vendors naturally handle.
Closing out the section, the panel discusses organizational enablement and workforce upskilling. Host James Belasco asks how organizations can empower non-specialist teams to understand and leverage customization techniques without overspending.
Karan stresses that democratizing AI does not require training an entire workforce of machine learning researchers. Existing cloud platforms and open-source infrastructure have already simplified the mechanics of model execution and fine-tuning. Instead, the most vital skill to cultivate across business teams is the ability to curate high-quality evaluation datasets. Teaching domain experts how to build representative eval sets allows teams to rigorously determine whether a problem actually requires full fine-tuning, or if lighter methods like prompt engineering, Retrieval-Augmented Generation (RAG), or prompt optimization are sufficient.
> "We don't necessarily need to upskill our workforce to be the machine learning researchers... if we can teach them on how to actually curate an highly good eval set, that can actually go a long way." — Karan Kothari
- **Primary Skills Focus:** Training domain experts in evaluation dataset curation over deep learning architecture engineering.
- **Customization Decision Tree:** High-quality evaluation sets help teams objectively choose between RAG, prompt optimization, parameter-efficient tuning, or full fine-tuning.
Here is a detailed, faithful walkthrough of the provided transcript segment:
[36:11](https://www.youtube.com/watch?v=TH6kW8KCDeQ&t=2171s) Centralizing AI Initiatives, Upskilling, and Organizational Strategy
Shruti Kamath highlights that upskilling the workforce is often the most challenging aspect of AI adoption within an organization. To address this, she advocates for centralizing AI initiatives and securing executive leadership sponsorship. Because AI technology is rapidly evolving and becoming embedded in daily operations, having dedicated teams focused on tracking the latest technological advancements helps determine where employees currently stand in their AI comprehension and what steps are needed to uplevel their skills.
At Mozilla, this effort is split strategically: Shruti’s team focuses on consumer-facing product use cases, while a separate internal organization dedicates itself to developer productivity by introducing tools like coding agents to streamline internal workflows. Implementing these enterprise tools requires significant coordination across legal and policy teams to establish proper usage guidelines. Furthermore, building internal "AI champions" across different domains fosters a culture of innovation. For example, a senior engineer at Mozilla recently researched strategies to address AI security risks—specifically targeting the "lethal trifecta," where untrusted content and prompt injections can exploit private datasets and leak sensitive information to malicious agents. The engineer developed a mitigation plan, demonstrating how creating space for innovation enables employees to propose new ideas and upskill the broader organization. The moderator reinforces this point, noting that connecting technical options directly to product constraints and collaborating with non-technical business users is crucial for building meaningful evaluation datasets and understanding deep business processes.
- **Key Security Risk:** The "lethal trifecta" refers to the threat combination where untrusted content and prompt injections exploit private datasets to transmit sensitive data to unauthorized external agents.
- **Mozilla's Organizational Structure:** Features a product-focused AI team for external users alongside an internal developer productivity team focused on coding agents and workflow efficiency.
[40:27](https://www.youtube.com/watch?v=TH6kW8KCDeQ&t=2427s) The Future of AI Customization, Model Distillation, and Open Weights
As the panel wraps up, the moderator poses a final question regarding the future of customization, asking whether organizations will lean heavily into fine-tuning open-weight models or continue relying on off-the-shelf base models. Karan Kothari advises starting with base models to establish a working proof of concept before attempting heavy customization. However, he notes a primary driver for fine-tuning: cost reduction at scale. When a base model requires an immense 120-billion parameter architecture to perform a simple task, organizations can generate high-quality examples and distill the capability down into a 4-billion parameter model to drastically reduce operational expenses.
Shruti Kamath agrees that off-the-shelf models, particularly when combined with Retrieval-Augmented Generation (RAG), will satisfy the vast majority of enterprise use cases as base capabilities improve. She emphasizes the growing power of open-source and open-weight models, referencing achievements like Llama 3.1 and community security responses. She highlights industry collaborations like the AI Alliance (initiated by Meta and IBM, with members including Nvidia and Mozilla) that champion open source. Rounding out the discussion, Edmund Clark shares how higher education handles this division: administrative operations rely on established frontier models to minimize risk, whereas academic research and teaching leverage open-weight models on platforms like the National Research Platform, allowing faculty and students to modify underlying parameters directly.
- **Model Distillation:** High-quality output examples from a 120B parameter model can be used to distill capabilities down into a 4B parameter model to cut deployment costs.
- **Enterprise vs. Research Split:** Enterprise and administrative workflows favor secure frontier models, while research institutions favor customizable open-weight models on infrastructure like the National Research Platform.
## Notable quotes
> [21:02](https://www.youtube.com/watch?v=TH6kW8KCDeQ&t=1262s) Instead of the burden becomes how can you sort of evaluate that synthetic data to make sure it really represents your business use case... that is a simpler way to go rather than trying to generate the eval dataset from ground. — Karan Kothari
> [27:06](https://www.youtube.com/watch?v=TH6kW8KCDeQ&t=1626s) If you just change the base model and go through the fine-tuning process again, you can significantly reduce your downstream cost of inference quite a lot. — Karan Kothari
> [34:08](https://www.youtube.com/watch?v=TH6kW8KCDeQ&t=2048s) We don't necessarily need to upskill our workforce to be the machine learning researchers... if we can teach them on how to actually curate an highly good eval set, that can actually go a long way. — Karan Kothari
## Mentioned
Edmund Clark, James Belasco, Karan Kothari, Shruti Kamath, California State University, Vanguard, Mozilla, Firefox, Lightning AI, Cursor, Reflection, National Research Platform, CENIC, Kimi, DeepSeek, T5-Flan, YOLO, vLLM, SGLang, OpenAI, ChatGPT, Houghton Mifflin Harcourt, Macmillan, AI Alliance, Meta, IBM, Nvidia, Llama 3.1
## Video description
Edmund Clark - Chief Information Officer, The California State University
James Belasco - Head of Data Science for Advice and Wealth Management, Vanguard
Karan Kothari - Principal Engineer, Lightning AI
Shruti Kamath - Sr Director, AI and Machine Learning, Mozilla
Generative AI is powerful out of the box, but its true enterprise value comes from tailoring models to specific organizational needs. This panel will explore the spectrum of customization strategies, from lightweight prompt engineering and retrieval-augmented generation (RAG) to full-scale fine-tuning and domain adaptation. Panelists will share real-world use cases where companies have successfully aligned frontier models with industry-specific data, compliance requirements, and customer contexts. The discussion will also cover emerging tools, platforms, and vendor approaches that make customization accessible while managing cost and complexity. Attendees will walk away with a clear understanding of how to choose the right customization path for their projects and maximize ROI from generative AI.
Mozilla 与 CSU 高管梳理企业 AI 定制化的成本陷阱
微调生成式 AI 仅在提升极端用例精度和模型蒸馏时具有效益，而长期部署成本以及与供应商功能的重叠，使得对大多数企业应用而言，完全自建定制模型令人望而却步。
与会嘉宾探讨了各机构在根据特定行业需求定制前沿模型和开源权重 AI 模型时，如何权衡性能、成本与复杂度。通过医疗推理模型、Firefox 本地智能标签页分组以及高等教育开放教材计划等具体案例，发言人概述了在模型选择、合成数据生成和推理优化方面的实用决策框架。他们强调应优先构建评估数据集而非过早进行微调，并警告切勿重复开发商业软件供应商已能提供的功能。
企业在定制生成式 AI 时，应在从提示工程、RAG 到全量微调的各层级方案中，权衡学生学习成效、隐私与成本。 (0:04)
对于行政运营，机构应尽量减少定制以降低成本、风险和合规阻力；而学术与科研场景则通常需要微调开源权重模型。 (4:13)
现成模型搭配零样本/单样本提示以及 RAG 即可解决大部分企业需求；微调应仅保留用于最后 5%–10% 的性能提升或单位成本优化。 (4:13)
利用来自更大型前沿模型的合成数据微调小型开源模型（例如将 T5-Flan 压缩至 60 MB 以下），能够实现快速且保护隐私的端侧 AI 功能，如 Firefox 的智能标签页分组。 (8:42)
尝试在内部自建复杂的 AI 工具可能导致成本激增甚至项目流产，尤其是当商业软件供应商极有可能推出类似现成功能时。 (14:47)
Reading summary
Enterprise AI customization requires a disciplined progression that prioritizes off-the-shelf prompting and Retrieval-Augmented Generation (RAG) before fine-tuning, which should be reserved for strict cost, latency, or edge-case constraints. Drawing on initiatives at Mozilla, Lightning AI, and the California State University system, panelists emphasize that long-term operational success depends on managing runtime serving expenses and establishing rigorous evaluation datasets.
Tiered Frameworks: Prompting, RAG, and Targeted Tuning (4:13)
Panelists advise organizations to treat model customization as a graduated ladder rather than an immediate mandate. Shruti Kamath of Mozilla notes that off-the-shelf models configured with zero-shot or one-shot prompting satisfy the vast majority of enterprise requirements, while RAG serves as the standard intermediate step for querying internal proprietary datasets. In this framework, fine-tuning is reserved for squeezing out the final 5% to 10% of domain performance or adjusting cost-quality balances across hardware layers.
Ed Clark outlines a bifurcated institutional policy at the California State University (CSU) system. For administrative workflows, CSU enforces a rule of minimal necessary customization to mitigate operational risk, compliance issues, and licensing costs. In contrast, academic research and instructional environments leverage shared infrastructure—such as the National Science Foundation–funded National Research Platform—to enable faculty and students to fine-tune open-weight architectures directly for specialized research.
Specialized Implementations and the Build-Versus-Buy Risk (8:42)
Targeted fine-tuning proves critical when proprietary reasoning or strict edge deployment constraints make frontier base models unviable. Karan Kothari highlights a healthcare client that fine-tuned models over clinical evidence to construct a diagnostic second-opinion tool, overcoming base models' inability to perform nuanced medical reasoning. Similarly, Mozilla developed its Firefox Smart Tab Grouping feature by fine-tuning a small T5-Flan model to an on-device footprint below 60 MB. By generating synthetic training labels using larger backend models and validating them with human evaluations, Mozilla doubled baseline performance while executing entirely on local user hardware.
Conversely, Ed Clark cautions against building in-house AI tools that compete with emerging commercial products. CSU attempted to develop an open-textbook AI tutor to counter commercial courseware that can cost students $400 per book. As the scope expanded across more than 200 textbooks and required complex pedagogical guardrails, customization expenses escalated, leading CSU to shelve the project. Clark warns enterprises against spending internal resources to build custom capabilities, such as automated OCR transcript extraction, when off-the-shelf document vendors are already integrating identical features.
Inference Economics and Total Cost of Ownership (18:33)
According to Karan Kothari, model serving at scale represents the true primary cost driver of enterprise AI, far exceeding the initial compute required for fine-tuning. Deployments must balance three operational knobs: quality, cost, and system control. Teams can optimize inference through a multi-layered approach: model-level techniques like quantization and model distillation (such as distilling a 120-billion parameter model into a 4-billion parameter alternative), serving-level techniques like speculative decoding and prompt caching, and specialized runtime engines like vLLM and SGLang.
Managing Total Cost of Ownership (TCO) also requires architectural adaptability. Because new high-performing open-weight models arrive every few months, re-running fine-tuning pipelines on newer, smaller base models can dramatically lower ongoing inference expenses. To maintain flexibility and avoid vendor lock-in, Kamath notes that Mozilla relies on pay-per-use vendor instances rather than multi-year reserved GPU commitments when piloting unproven products, preserving strategic agility until usage patterns mature.
Data Enablement, Upskilling, and System Governance (34:08)
Democratizing AI within an enterprise hinges on data curation rather than teaching deep learning engineering. Kothari argues that the single most vital skill to cultivate among domain experts is the creation of representative evaluation datasets, which allow teams to objectively determine whether an application requires fine-tuning, RAG, or basic prompt optimization. Synthetic data generation, derived from a small set of high-quality human examples and verified by human raters, enables organizations to overcome data scarcity while protecting proprietary user privacy.
At the organizational level, scaling AI requires structural alignment and rigorous security protocols. CSU unified 22 independent university campuses under a shared data platform and standardized technology guidelines to prepare its infrastructure for enterprise AI. At Mozilla, internal adoption is divided between customer-facing product teams and developer productivity units implementing coding assistants. To mitigate emerging risks like the 'lethal trifecta'—where prompt injections and untrusted inputs interact with private datasets to leak data externally—organizations must establish cross-functional governance across engineering, policy, and legal departments.
The panel concludes that while base models paired with RAG will handle most business workflows, fine-tuning remains indispensable for low-footprint local inference and deep computational cost reduction at scale. The primary challenge moving forward is organizational: enterprises must cultivate robust evaluation benchmarks and avoid over-engineering custom models for problems that commercial vendor platforms are poised to commoditize.
## Background
本内容由 Ai4 于 2026 年 8 月 21 日发布在 YouTube 平台上。视频记录了一场关于生成式 AI 定制化与微调策略的企业级圆桌论坛，汇聚了来自加州州立大学、先锋领航（Vanguard）、Lightning AI 和 Mozilla 的技术高管，深入探讨了从轻量级 Prompt 工程、RAG 到全量微调的选型路径与真实落地案例。
## TL;DR
圆桌讨论围绕企业在大模型应用中如何在开箱即用、RAG 和模型微调之间做权衡展开。嘉宾们结合医疗临床推理、Mozilla 端侧智能标签分组以及加州州立大学的教材导师等实际案例，指出微调的核心开销往往在于长期的在线推理（Serving）而非单纯的训练。专家建议企业优先构建高质量的业务评估数据集（Eval Set），通过前沿大模型合成与蒸馏数据，并在必要时持续微调更轻量级的开源基座模型以大幅压缩 TCO。同时，组织需建立中心化治理与安全防御机制，避免盲目自研供应商已有的成熟方案。
## Atmosphere & style
整场圆桌对话务实、严谨且充满一线工程实践的真知灼见，没有浮夸的技术炒作。四位来自学术界、工业界和底层基建的技术领袖互动默契、交流坦诚，既有对技术前沿与开源生态的热情展望，也有对算力账单失控、项目折戟教训的冷静复盘。听众能清晰感受到一种在创新冲动与商业成本控制之间反复权衡的工程理性，节奏紧凑且干货密集。
## Key takeaways
- [04:13](https://www.youtube.com/watch?v=TH6kW8KCDeQ&t=253s) 大模型定制化应遵循“最小定制量原则”，大部分常规需求优先通过 Prompt 工程与 RAG 解决，仅在需要榨取最后 5%~10% 的质量与成本极限时才投入微调。
- [00:04](https://www.youtube.com/watch?v=TH6kW8KCDeQ&t=4s) Mozilla 布局端侧本地模型策略，核心聚焦隐私安全与用户自主权，确保用户私有数据完全不出设备端并支持自由切换开源模型。
- [08:42](https://www.youtube.com/watch?v=TH6kW8KCDeQ&t=522s) 针对高壁垒、私有化的专有领域数据（如医疗临床推理），采用“RAG 检索事实证据 + 微调注入领域推理逻辑”的双轨架构是打造高可靠 AI 辅助工具的必经之路。
- [11:49](https://www.youtube.com/watch?v=TH6kW8KCDeQ&t=709s) Firefox 标签页智能分组实践证明：利用前沿大模型离线生成高质量合成样本，将能力微调蒸馏至 60MB 以下的轻量小模型（如 T5-Flan），可在端侧实现性能翻倍且兼顾极低算力消耗。
- [14:47](https://www.youtube.com/watch?v=TH6kW8KCDeQ&t=887s) 警惕过度定制与重复造轮子：通用基座配合技能与知识库可满足绝大多数日常业务，非核心通用功能应优先采购成熟供应商方案，避免因系统维护陷入高昂沉没成本。
- [18:22](https://www.youtube.com/watch?v=TH6kW8KCDeQ&t=1102s) 模型微调算力仅是冰山一角，长期在线推理部署（Serving）才是主要的成本黑洞；选型时应基于部署平台的延迟、吞吐 SLA 约束倒推基座模型，并尽早锁定评测集以防算力浪费。
- [21:54](https://www.youtube.com/watch?v=TH6kW8KCDeQ&t=1314s) 数据构建最敏捷的工程实践是“前沿大模型合成 + 人工严格校验 + 数据增强”，既能低成本解决冷启动数据稀缺问题，又能规避直接抓取用户隐私数据的合规风险。
- [24:03](https://www.youtube.com/watch?v=TH6kW8KCDeQ&t=1443s) 控制大模型长期推理成本需跨技术栈协同：覆盖模型层（量化与蒸馏）、服务层（智能缓存与推测解码）以及内核硬件层（定制算子与适配 GPU 选型）。
- [27:06](https://www.youtube.com/watch?v=TH6kW8KCDeQ&t=1626s) 通过“持续再微调（Re-fine-tuning）”降低 TCO：开源基座迭代迅速，保持评估集稳定并定期将权重迁移至更新、更小、吞吐更高的新模型，其长远推理节约远超重新微调的开销。
- [29:20](https://www.youtube.com/watch?v=TH6kW8KCDeQ&t=1760s) 在未验证产品市场契合度（PMF）的探索性新场景中，应坚决避免供应商锁定，宁愿短期承受稍高的按需付费（Pay-per-use）单价，以灵活性换取验证空间。
- [34:08](https://www.youtube.com/watch?v=TH6kW8KCDeQ&t=2048s) 推动组织 AI 能力联邦化的最高 ROI 路径并非算法底层自研，而是促成业务人员与工程师协同构建精准映射业务评测基准的高质量评估数据集（Eval Set）。
- [36:26](https://www.youtube.com/watch?v=TH6kW8KCDeQ&t=2186s) 建立中心化战略指导与业务线 AI 领航员（AI Champions）协同机制，重点防御不受信外部输入、提示词注入与私有数据意外泄露构成的“致命三要素”。
- [40:27](https://www.youtube.com/watch?v=TH6kW8KCDeQ&t=2427s) 未来定制化演进将以“先用前沿大模型验证业务，再蒸馏至 4B 左右小模型规模化降本”为主线，并在组织内部形成商业合规大模型与开源自主权重并存的双轨治理范式。
## Detailed walkthrough
### [00:04](https://www.youtube.com/watch?v=TH6kW8KCDeQ&t=4s) 嘉宾介绍与定制化战略动因
论坛在先锋领航（Vanguard）顾问与财富管理数据科学主管 James Belasco 的主持下开场。James 首先邀请各位与会嘉宾自我介绍，并分享各自机构探索大模型定制化的战略目标、核心诉求及启动动因。加利福尼亚州立大学（California State University, CSU）首席信息官 Ed Clark 首先发言，他指出加州州立大学系统是全美规模最大的高等教育系统，全美每 20 位大学毕业生中就有 1 位出自该校。面对如此庞大的群体，CSU 考量技术定制化的唯一核心指标就是“对学生的实质成效”——如何通过技术手段帮助学生提高学业成功率、按期毕业并顺利实现就业。
来自 Mozilla 的 AI 与机器学习负责人 Shruti Kamath 随后分享了她在该领域的深厚背景。她投身 AI/ML 已逾十年，职业生涯始于亚马逊最早的深度学习团队担任算法工程师，其后在 Zillow、PayPal、Chewy 等企业担任工程管理要职。谈及 Mozilla 与 Firefox 浏览器的定制化理念，Shruti 将其归纳为三大战略支柱：
1. **可信度、透明性与隐私安全**：Mozilla 重点布局本地端侧 AI（on-device models），确保用户数据无需通过网络传输即可在设备端完成计算，杜绝数据泄露与越权访问隐患。
2. **模型选择权与用户自主性**：不同于主流浏览器将用户绑定在单一封闭生态中的做法，Mozilla 提倡让用户拥有自主权，支持用户自带模型（Bring Your Own Model）或灵活选用开源模型组合。
3. **成本与质量的动态权衡**：在模型表现与推理成本之间寻找最佳平衡点，通过针对性定制实现性价比最优。
最后，Lightning AI 首席工程师 Karan Kothari 介绍了他们作为新型云基础设施提供商的定位。Lightning AI 目前不仅为 Cursor、Reflection 等前沿 AI 工具提供用于模型预训练的底层 GPU 算力集群，同时也面向企业客户提供全流程的机器学习工作流微调与部署方案。在模型定制层面，他们的核心目标是为企业赋能，提供从轻量级参数高效微调（PEFT）到强化学习（RL）环境的全套工具链，让客户能够根据具体场景选取最匹配的工具实现业务目标。
- 加利福尼亚州立大学（CSU）是全美最大大学系统，全美 5%（1/20）的本科毕业生来自该系统。
- Mozilla 的端侧模型策略聚焦本地隐私保护，数据完全不下发网络、不出设备端。
- Lightning AI 支撑了 Cursor 和 Reflection 等头部 AI 原生产品的预训练与微调算力需求。
---
### [04:13](https://www.youtube.com/watch?v=TH6kW8KCDeQ&t=253s) 决策框架：从提示词、RAG 到模型微调的权衡
针对企业在评估业务场景是否需要进入微调阶段时应遵循的决策框架，嘉宾们展开了多维度的拆解。Ed Clark 从高校运营的双重属性切入，将场景清晰地划分为“行政管理侧”与“学术研究侧”。在行政运营管理方面，CSU 坚守的核心法则是“以达成业务目标所需的最小定制量为原则（do the least amount of customization）”。Ed 强调，切忌一上来就盲目投入模型微调，团队应当首先审视当前方案为何无法奏效，明确实现这一业务目标愿意承受多大的成本，并充分评估背后的合规风险与安全边界。而在学术与科研侧则是截然相反的景象：得益于加州国家研究平台（National Research Platform）以及 UC 系统与 CSU 系统共享的 CENIC 加州 AI 计算资源，师生们能够自由调用 DeepSeek、Kimi 等开源权重模型（open-weight models），为了达到特定前沿科研课题的目标，针对这些模型进行深度微调往往是必经之路。
```mermaid
flowchart TD
accTitle: 模型定制化阶梯演进图
accDescription: 展示从开箱即用到提示词工程、检索增强生成、微调直至底层硬件优化的技术路径选择
subgraph L1["轻量探索层 (Low Cost & Effort)"]
A["开箱即用 / 通用基础模型"] --> B["提示词工程 (Zero-shot / Few-shot)"]
end
subgraph L2["上下文增强层 (Context Augmentation)"]
B --> C["检索增强生成 (RAG)\n引入专有工作流与私有知识库"]
end
subgraph L3["模型深度定制层 (Model Fine-Tuning)"]
C --> D["参数高效/全量微调 (PEFT / SFT)\n榨取最后 5%~10% 的质量与成本极限"]
end
subgraph L4["底座与系统优化层 (System Optimization)"]
D --> E["内核与硬件层定制 (Kernel & Hardware Optimization)"]
end
```
Shruti Kamath 从工程技术演进的角度给出了分层演进的递进逻辑。她认为大部分现成的基础模型在零样本（zero-shot）或单样本（one-shot）提示工程下就已经具备相当出色的泛化能力；当企业手头坐拥大量组织内部特有的专有数据和业务流（例如未被互联网通用语料覆盖的内部系统运作方式）时，第一步应当首选 RAG（检索增强生成）系统来检索并注入背景上下文。只有当业务进入深水区，需要为了极端严格的质量标准或降低推理成本去“榨取最后 5% 到 10% 的边际收益”时，才真正需要动用微调手段。Shruti 特别补充道，广义的定制化绝不仅限于模型权重本身，向下延伸还涵盖了算子内核（kernel）与底层硬件层面的专门优化，最终的决策必然是性能、质量与成本三者之间的权衡博弈。
---
### [08:42](https://www.youtube.com/watch?v=TH6kW8KCDeQ&t=522s) 落地案例：医疗临床推理的 RAG 与微调实践
在主持人引导各方分享微调的实际落地案例时，Karan Kothari 深入剖析了 Lightning AI 平台上一家医疗健康客户的真实实践。该客户坐拥数 PB 级别高度专有的临床推理数据，其中涵盖了全球顶尖临床医生在面对复杂病例时逐步推导演绎出临床诊断结论的思维样本。此前，无论是直接调用前沿闭源旗舰大模型（frontier models），还是部署开源大参数模型，在面对这套高难度专有医学逻辑时推理准确率均表现惨淡，模型根本无法自主复现医生的严密推理路径。
针对此类通用大模型“从未见过”的私有高壁垒数据，该团队采取了组合架构策略：一方面利用 RAG 从海量数据库中检索患者病例证据并精准呈现给模型，另一方面利用专有临床决策链数据对基础模型进行深度微调，专攻领域特定的推理逻辑（domain-specific reasoning）。最终，这套模型被成功打造成协助主治医生做出判断的“AI 临床伴侣”，充当医生诊断时的辅助第二意见提供者（second opinion）。
针对主持人关于“此类重度投资微调是否真正必要、ROI 从何而来”的追问，Karan 强调这完全是由问题本身的专有属性所决定的。由于前沿大模型实验室根本无法获取这部分涉及极高隐私与专业度的封闭医疗数据，现有通用基座模型“开箱即用”的状态完全无法满足实际医疗诊断的要求。对于企业而言，要打造医生真正愿意信赖的生产力产品，这种前期的高成本技术投入是不可绕过的“信仰跃迁（leap of faith）”。同时，随着底层开源基座模型自身推理能力的持续迭代，建立在微调之上的专有能力会随着底座的升级而释放出更显著的长期复利。
- 该医疗案例处理的专有临床推理数据体量达到 PB（Petabyte）级别。
- 架构上采用“RAG 抓取证据 + 微调赋予推理能力”的双轨方案，而非单靠检索或单靠微调。
- 最终形态并非替代医生，而是作为辅助决策的专业级伴随工具（clinical companion）。
### [11:25](https://www.youtube.com/watch?v=TH6kW8KCDeQ&t=685s) 微调是一个持续迭代的过程
微调绝非一劳永逸的单次任务，而是一个需要不断试验与优化的迭代周期。随着开源生态与学术界的快速演进，每隔几个月就会涌现出性能更优的基础模型以及更高效的微调新技术，团队可以不断尝试将其应用到业务目标中。
主持人随后指出，这种持续维护和迭代势必会带来长期的维护成本，但在深入探讨成本之前，圆桌将先转向具体的企业落地案例分享。
```mermaid
flowchart LR
accTitle: 迭代式微调生命周期
A["业务目标与需求定义"] --> B["选择基础模型与微调技术"]
B --> C["训练微调与离线评估"]
C --> D["端侧/线上部署与监控"]
D --> E{"新模型/新技术发布？"}
E -- "是" --> B
E -- "否" --> D
```
---
### [11:49](https://www.youtube.com/watch?v=TH6kW8KCDeQ&t=709s) Mozilla Firefox 的端侧微调实践：智能标签页分组
Mozilla 机器学习高级总监 Shruti Kamath 分享了 Firefox 浏览器中的一项创新功能——“智能标签页分组”（Smart Tab Grouping）。该功能能够将用户浏览的相关网页标签自动归类，并生成贴合语义的群组名称。整个推理过程完全依托端侧（On-Device）模型在本地运行。
在群组名称生成的场景中，团队选择专门微调一个小模型，主要出于两大考量：
1. **语义泛化度过高的问题**：开箱即用的通用大模型给出的标签名过于宽泛（例如简单标注为“旅行”）。而用户的实际分组意图极度依赖具体情境，可能是一次与朋友共同规划的旅行，或是某项具体活动，必须通过微调赋予模型生成精细化、场景化命名的能力。
2. **硬件与端侧兼容性要求**：为了确保该功能能在各种配置悬殊的终端硬件上流畅运行（涵盖不同的内存限制、CPU 及 GPU 规格），模型体积必须极其轻量化。
```mermaid
flowchart TD
accTitle: Mozilla 端侧微调与数据生成流
subgraph Offline ["离线数据处理与蒸馏"]
G["通用大模型 (Frontier/Big Models)"] -->|"批量生成高质量标注与微调样本"| D["精标训练集"]
end
subgraph OnDevice ["端侧训练与部署"]
D -->|"监督微调 (Fine-Tuning)"| T["T5-Flan 模型 (<60MB)"]
T -->|"部署到端侧 (Firefox)"| C["多样化终端硬件 (不同 CPU/GPU/内存)"]
C -->|"本地推理"| R["精准/场景化的标签页分组名称"]
end
```
Mozilla 最终选择微调相对成熟但极其轻量的 **T5-Flan** 模型，整体参数体积压缩至 **60 MB 以下**。相较于直接调用开箱即用的通用模型，微调后在质量与性能表现上实现了“两倍以上的飞跃”，且几乎没有肉眼可见的质量回退。对于大型前沿模型，Mozilla 将其定位于离线标注和数据生成环节，利用大模型打标后的数据训练小模型，达成了性能与端侧算力的平衡。
---
### [14:47](https://www.youtube.com/watch?v=TH6kW8KCDeQ&t=887s) 加州州立大学的教训与案例：开源教材 AI 导师与智能制造
加州州立大学（California State University）首席信息官 Edmund Clark 坦诚分享了一个在定制化过程中遭遇失败的典型项目，以及一个工业界微调的成功样本：
- **大模型与用户需求之间的鸿沟**：当前通用基础模型的能力极其强大，而在加州州立大学系统中，95% 的业务需求仅需结合基础模型、RAG（检索增强生成）以及技能文件（Skills File）就能解决。
- **软件采购准则**：“永远不要去自行研发供应商已经在做或即将推出的产品，能买现成的就买现成的，避免重复造轮子。”
```mermaid
flowchart TD
accTitle: 开源教材 AI 导师项目成本失控与失败归因
A["初衷：推广开源教材，为学生节省数亿美元"] --> B["教师需求：严格限制在单本教材内答疑"]
B --> C["瓶颈：单本教材语料不足，教学效果不佳"]
C --> D["过度定制：引入200+本教材，嵌入认知失调等复杂教学法"]
D --> E["成本爆炸：算力与定制开销超出单个大学承担能力"]
E --> F["试图组建高校联盟失败，项目终止"]
```
#### 失败案例：开源教材智能导师（Open Textbook Tutor）
为缓解贫困学生（如 Pell Grant 受助生）高昂的教材支出负担，校方启动了开源教材 AI 导师项目，旨在支持多语言并覆盖全学科。然而在落地过程中问题频发：
1. 最初教师要求“严格将回答限制在指定教材内”，但少量文本不足以支撑良好的交互表现；
2. 团队被迫不断扩充教材库（最终达到 200 多本），并尝试对模型交互进行深度定制，要求其不仅仅直接给出答案，还要引发学生的“认知失调”（Cognitive Dissonance）以达到真正的教学目的；
3. 系统的深度定制导致成本急剧飙升，在未能成功联合其他大学组建联盟分摊费用的情况下，单校已无力支撑如此高昂的研发与算力成本。
#### 成功案例：智能制造中的 YOLO 计算机视觉微调
与之相对，加州州立大学一位专注于智能制造的教师在使用开源图像检测模型 **YOLO (You Only Look Once)** 时，遇到了模型无法开箱识别特定工厂车间布局与工件的问题。通过在车间环境中采集定制图像进行反馈微调，并对模型输出参数与格式进行适配，成功产出了符合学术论文与实际产线精度标准的模型。
---
### [18:22](https://www.youtube.com/watch?v=TH6kW8KCDeQ&t=1102s) 模型微调与部署的成本决策框架
微调的整体开销往往被低估。Lightning AI 的首席工程师 Karan Kothari 强调：**微调本身的算力成本只是冰山一角，模型在线提供服务（Model Serving）的长期推理成本才是主要的资金黑洞**，在大规模高并发场景下尤其显著。
```mermaid
flowchart LR
accTitle: 降本决策与候选模型收敛路径
subgraph Constraints ["第一公里：明确平台与服务 SLA 约束"]
P["推理平台架构限制"]
S["吞吐量与延迟 SLA 指标"]
end
subgraph Selection ["模型选择与收敛"]
Constraints -->|"过滤不合格候选"| F["大幅收敛的基础模型候选集"]
end
subgraph QualityGate ["最后一公里：锁定高标准评估体系"]
F --> E["构建确定且固定的评测集 (Strong Eval Set)"]
E -->|"避免重复无效微调"| O["高效微调与上线推理"]
end
```
Karan 提出了评估微调与部署的“三要素旋钮”（Three Knobs）与降本工作流：
| 评估维度 | 核心考量因素 | 对成本与架构的影响 |
|:--- |:--- |:--- |
| **质量 (Quality)** | 业务可接受的最低精度与感知质量底线 | 决定模型参数量大小以及是否需要多阶段微调 |
| **成本 (Cost)** | 包含单次微调预算与长期的推理 Serving 账单 | 模型体积越大、并发越高，Serving 成本呈指数级上升 |
| **控制力 (Control)** | 基础设施掌控度、数据私密性与输出确定性 | 决定是采用云端托管 API 还是自建端到端开源堆栈 |
为了避免盲目微调造成的算力浪费，Karan 建议：
- **从 Serving 平台倒推模型选型**：在决定微调前，优先审视自身的模型部署平台约束（支持的算子、硬件架构）以及吞吐量（Throughput）和延迟（Latency）的 SLA 目标，从而在海量候选模型中快速筛选出极少数达标的基座模型。
- **锁定评测集（Eval Set）**：尽早锁死评测数据集。如果评估标准在开发过程中频繁变动，团队将被迫不断重新微调和跑测，带来巨大的隐性研发与算力浪费。
---
### [21:14](https://www.youtube.com/watch?v=TH6kW8KCDeQ&t=1274s) 微调数据获取策略与前沿模型的蒸馏价值
在微调链路中，高质量数据的获取和清洗往往是最大瓶颈。当被问及企业如何构建足量的微调数据时，Karan 指出：
- 企业内部的私有专有数据（Proprietary Data）以及明确的业务成效记录是天然的切入点。
- 仅靠人工标注来生成微调所需的大规模高质量样本极其耗时费力。
- 当前最主流、最高效的工程实践是**利用前沿大模型（Frontier Models）进行数据蒸馏与合成**：向大模型输入少量高标准的 Few-shot 样例，利用其泛化能力批量生成训练数据，再将其用于小模型的监督微调，从而在极低成本下快速达成高质量数据集的构建。
### [21:54](https://www.youtube.com/watch?v=TH6kW8KCDeQ&t=1314s) 合成数据与隐私合规：从标签生成到微调规模化
在模型微调与评估的数据构建阶段，合成数据（Synthetic Data）扮演了关键角色。Karan Kothari 指出，与其从零开始艰难收集真实的评估数据集，不如利用大模型合成大部分数据，随后将核心精力转向“如何严格评估这些合成数据，确保其真实贴合业务场景”。这种方式在工程实践中往往更为敏捷和高效。
来自 Mozilla 的 Shruti Kamath 对此深有同感，并结合 Mozilla 的产品哲学补充了这一做法的必要性。Mozilla 极其注重用户隐私保护，严格杜绝使用用户私有数据进行训练。为了解决数据稀缺问题，团队利用前沿基础模型（Frontier Models）生成初始标签，经过人工评估（Human Evals）进行严格校验与筛选后，再结合数据增强（Data Augmentation）技术进行扩展，以此满足微调所需的庞大数据量。Shruti 举例道，在开发浏览器的“智能标签页分组”（Smart Tab Grouping）功能时，团队正是通过这套“模型合成 + 人工清洗 + 规模化增强”的流程，顺利达成了微调所需的高质量数据规模。
---
### [23:01](https://www.youtube.com/watch?v=TH6kW8KCDeQ&t=1381s) 打破高校数据孤岛：加州州立大学的统一数据战略
加州州立大学（California State University, CSU）首席信息官 Edmund Clark 分享了高校体系在数据治理上的独特挑战。CSU 系统规模庞大，下辖 22 个独立校区，拥有 22 位校长和 22 位首席信息官。长期以来，各个校区习惯于在各自的数据孤岛（Siloed Pockets）中独立运作，甚至对基础专业（Majors）的定义和数据标准都各行其是，导致整体系统极度碎片化。
改变的契机来自于新任总校长（Chancellor）提出的宏大目标——明确要求大幅提高全系统的毕业率并改善毕业生的就业表现。如此清晰且激进的目标倒逼整个系统必须步调一致。Edmund 指出，全校达成共识：“我们深知未来将由 AI 驱动，但要赋能这个未来，必须先把数据底座搞对。”为此，CSU 开始推行统一数据平台，并制定了一套技术与实施指南（Cookbook），为在整个系统内解决深层次业务问题筑牢根基。
---
### [24:03](https://www.youtube.com/watch?v=TH6kW8KCDeQ&t=1443s) 推理成本的多层优化：模型量化、推测解码与定制内核
关于大语言模型的成本控制，Shruti Kamath 强调，训练成本固然存在，但模型上线后的推理（Inference）阶段才是成本极易失控的“重灾区”。面对定制化模型带来的推理开销，她提出了跨越技术栈的多层优化框架：
```mermaid
flowchart TD
accTitle: LLM推理阶段多层成本优化架构
subgraph ModelLayer["模型层优化 (Model Layer)"]
M1["模型量化 (Quantization)"]
M2["模型蒸馏 (Distillation)"]
end
subgraph ServingLayer["服务层优化 (Serving Layer)"]
S1["智能缓存 (Caching)"]
S2["推测解码 (Speculative Decoding)<br/>小模型起草 + 大模型验证"]
end
subgraph KernelLayer["底层与硬件优化 (Kernel & Hardware Layer)"]
K1["定制计算内核 (Custom Kernels)"]
K2["适配业务特征的硬件与 GPU 选型"]
end
ModelLayer --> ServingLayer --> KernelLayer
```
- **模型层（Model Layer）**：采用量化（Quantization）与知识蒸馏（Distillation）技术，大幅压缩模型体积，在尽量减少性能损失的前提下降低单次推理算力消耗。
- **服务层（Serving Layer）**：引入完善的缓存机制（Caching），并采用推测解码（Speculative Decoding）策略——通过小型轻量模型先生成草稿词元，再由大模型快速并行验证，使大部分计算落在低成本的小模型上。
- **内核与硬件层（Kernel & Hardware Layer）**：针对具体业务数据和场景开发定制化计算内核（Custom Kernels），同时精细化选配 GPU 和硬件架构，以实现最大化的能效与吞吐量。
针对量化技术，Karan Kothari 补充了一个工程落地中的重要教训：量化虽然只带来轻微的质量损失，但选择候选模型时必须非常谨慎。如果业务场景对质量下降毫无容忍空间（Zero Wiggle Room），后续通过量化来做成本优化的操作空间就会被严重压缩。
---
### [26:10](https://www.youtube.com/watch?v=TH6kW8KCDeQ&t=1570s) 从“随意使用”到按量计费：学术界的算力账单冲击
谈及成本核算，Edmund Clark 坦言高等教育机构目前正经历一段有趣的过渡期。此前，CSU 签署了全校范围的 ChatGPT 机构协议，各部门和师生基本处于“想怎么用就怎么用”的自由状态。然而，随着 OpenAI 开始严格按照消耗的积分点数（Credits）进行实际计费，学校内部不得不紧急踩下刹车并重新审视预算，甚至出现了“某位研究人员突然严重超额”的窘境。
而在科研算力方面，得益于美国国家科学基金会（NSF）资助的国家研究平台（National Research Platform, NRP），CSU 在加州本地集群中部署了大约 1,000 张 GPU，供研究人员直接申请算力进行学术探索，暂时免受商业云端高昂账单的困扰。但面对企业级的大模型应用，学术界也正迅速意识到总体拥有成本（TCO）核算的紧迫性。
---
### [27:06](https://www.youtube.com/watch?v=TH6kW8KCDeQ&t=1626s) 总体拥有成本（TCO）与模型迭代：通过“持续再微调”降低长期推理支出
主持人 James Belasco 引出了企业面临的核心痛点：一旦选择微调，企业就必须承担微调模型的生命周期维护、硬件托管、基础设施运维等一整套持续支出。在不断变化的技术生态中，企业该如何权衡总体拥有成本（Total Cost of Ownership, TCO）？
Karan Kothari 给出了一个看似反直觉却收益显著的策略：**主动、持续地用最新的基础模型重新进行微调（Re-fine-tuning）**。他的核心逻辑在于：
1. 开源基础模型（Open-weight Models）每隔几个月就会迎来一次技术跃升，新模型往往体积只有原来的一半，而推理智能水平几乎没有衰减；
2. 由于主要的模型支出集中在下游的长期推理上，只要评估数据集保持稳定，仅需花费极少的重新微调成本，将权重迁移到更新、更小的开源底座上；
3. 新一代底座模型原生兼容高性能推理容器（如 vLLM、SGLang 等），能带来吞吐量的大幅飞跃，从而在推理端成倍降低总体硬件持有成本。
- 持续微调的投入远小于长期维持老旧、庞大模型所产生的高昂推理账单。
- 利用已有的验证集持续迁移新基座，是企业对抗技术过时与算力浪费的最优解。
---
### [29:20](https://www.youtube.com/watch?v=TH6kW8KCDeQ&t=1760s) 应对未经验证的新场景：避免供应商锁定与按需付费策略
Shruti Kamath 进一步将 TCO 划分为构建成本、运行成本、维护成本以及合规成本，并强调 TCO 的策略必须严格取决于业务场景的成熟度——是“完全从零孵化的新场景（Net-new Use Cases）”还是“已有业务上的增量优化（Incremental Use Cases）”。
针对尚未验证产品市场契合度（PMF）的探索性场景（如 Mozilla 在浏览器中新推出的 AI 助手“智能窗口” Smart Window），Shruti 建议早期必须保持高度敏捷，核心原则是**坚决避免供应商锁定（Vendor Lock-in）**：
- **按需付费优先于长期承诺**：尽管签订 3 到 5 年的 GPU 长期包月/采购协议单价更便宜，但对于新场景，宁愿在短期内承受稍高的按量付费（Pay-per-use）单价，也要保留随时切换架构与供应商的灵活性。
- **以灵活性换取验证期**：先通过弹性服务测算出真实的调用规模、并发峰值以及需要微调的数据体量，待业务成熟后再进行长期的基础设施投资与精确的 TCO 测算。
Edmund Clark 随后以教材出版商为例呼应了这一动态。诸如霍顿·米夫林·哈考特（Houghton Mifflin Harcourt）和麦克米伦（Macmillan）等大型出版巨头，正开始在其教科书中捆绑额外的 AI 辅导组件并加收费用，这种外部商业模式的演变也正直接影响着高校在自研方案（如 CSU 暂时搁置的 Student Companion 伴学助手项目）与外部采购之间的经济账权衡。
### [32:32](https://www.youtube.com/watch?v=TH6kW8KCDeQ&t=1952s) 高昂教材成本、教学落地脱节与“自建 vs. 采购”陷阱
Edmund Clark 以具体的数据和现实痛点切入讨论：他自己授课所用的两本教材，每本单价高达约 400 美元，这对于在校学生而言是一笔极为沉重的经济负担。从理念上看，高校若能利用 AI 技术为学生提供免费的数字化智能教材、配套辅导机制以及丰富资源，无疑是一个极具吸引力且激动人心的愿景。然而现实落地的核心阻碍在于，教师不可能在基础设施尚不健全、工具未完全成型的情况下贸然接入教学。技术团队往往仍处于方案的试点（pilot）阶段，但任课教师身处第一线教学现场，迫切需要开箱即用、高度稳定且功能完整的成品系统。这种试点探索与实际课堂需求之间的“交付时间差”与落地鸿沟非常明显。
基于此，Edmund 提出了极为务实的忠告：“不要试图去复刻外部供应商即将推出的通用功能”（"don't try to replicate what vendors are going to do"）。他举了一个高校行政中的典型痛点：目前依然有海量纸质成绩单流转，许多人便提议自研一套 AI 视觉扫描与自动成绩录入系统。但在现实中，几乎全球所有的主流文档管理软件公司（Document Management Companies）都在全力攻关同类功能。除非自研方案能够证明极其庞大的成本节约，并且组织自身有能力承担长期的系统迭代、升级维护与技术支持，否则盲目自建只会陷入高额维护成本（TCO）的深坑。对于非纯科技企业而言，与供应商成熟方案对齐往往比闭门造车更为明智。
- **高昂教育成本痛点**：单本教材价格约 400 美元，AI 辅助教学与免费智能教材具备极高社会与经济价值。
- **教学场景交付断层**：技术研发处于 Pilot 迭代期，而一线教师教学要求 100% 稳定的即时交付。
- **自研防踩坑原则**：避免投入资源复刻文档数字化等通用厂商赛道，审慎评估后期升级与维护成本。
---
### [34:08](https://www.youtube.com/watch?v=TH6kW8KCDeQ&t=2048s) 组织赋能与技能升级：评估集构建优于盲目算法研发
讨论随后转向组织层面的能力建设：各团队往往容易直接调用基础模型（Foundation Models），却缺乏微调（Fine-Tuning）与定制化意识，如何推动全组织的 AI 能力联邦化（Federate）并完成人才技能升级？Lightning AI 的 Karan Kothari 指出，企业并不需要把全体员工重新培养成底层机器学习研究员（Machine Learning Researchers）。在微调与模型定制的链路中，最核心且最具杠杆效应的技能是——**构建高质量的评估数据集（Eval Set）**。
Karan 阐述，模型训练与推理的基础设施在当今工具链下已高度标准化、平民化，真正的瓶颈在于能否产出精准映射业务用例的评测基准。只有掌握了高质量评测集的构建能力，团队才能清晰界定是否真正需要定制化：究竟是简单的 Prompt 优化、RAG（检索增强生成）就能达标，还是必须介入参数微调。主持人在总结中进一步强化了该观点，指出构建业务评测集需要非技术背景的业务人员（Business Users）与技术工程师深度协同，由业务人员将隐性的工作流程深度与业务判定标准沉淀为评测规则，这才是最具投资回报率的组织赋能路径。
```mermaid
flowchart TD
accTitle: AI 定制化选型决策流
accDescription: 展示从构建评测集出发，在 Prompt 工程、RAG 与模型微调之间做选择的决策流程。
subgraph Phase1["1. 关键组织能力"]
A["业务团队与技术团队协同"] --> B["构建高质量评估数据集<br/>(Business Eval Set)"]
end
subgraph Phase2["2. 方案效果与成本验证"]
B --> C{"基线评估与路径选型"}
C -- "轻量达标" --> D["Prompt 优化"]
C -- "依赖外部私有知识" --> E["RAG 检索增强生成"]
C -- "特定任务/降本/严格格式" --> F["参数微调 / 模型蒸馏"]
end
subgraph Phase3["3. 下游工程落地"]
F --> G["依托成熟基础设施<br/>自动执行训练与推理部署"]
end
```
---
### [36:26](https://www.youtube.com/watch?v=TH6kW8KCDeQ&t=2186s) 平台集中化赋能与安全文化：破解“致命三要素”
Mozilla 的 Shruti Kamath 强调了中心化赋能与高管赞助（Leadership Sponsors）对于组织推行 AI 的必要性。随着 AI 快速渗透到日常业务且演进日新月异，中心化的专属团队能够评估组织当下的 AI 理解阶段，制定分层跃迁路径，同时打通法务、合规与数据安全策略。以 Mozilla 为例，内部采用了清晰的双轨机制：一个组织专注于面向外部消费者的产品功能开发，另一个组织则专注于内部研发效能（Developer Productivity），系统性引入 Coding Agents 优化研发工作流。
Shruti 还分享了在组织内部建立“AI 领航员/布道师”（AI Champions）机制的经验。针对前沿安全风险，Mozilla 资深工程师深入研究了 AI 安全领域的防范策略，重点防御所谓的**“致命三要素”（Lethal Trifecta）**：
1. **不受信任的外部内容输入（Untrusted Content）**；
2. **潜在的提示词注入漏洞（Prompt Injections）**；
3. **私有数据意外泄露（Private Data Exfiltration）**——即敏感数据被不受控的 AI 智能体传输至恶意攻击端（Evil Agent）。
通过由资深专家研发防御方案并向全组织同步，Mozilla 建立起了一种鼓励创新提案与安全规范并行的组织文化，让跨领域的团队都能在合规安全的轨道上采纳最新技术。
- **组织治理架构**：中心化战略指导 + 业务线领航员（AI Champions）协同发力。
- **Mozilla 落地分工**：外部产品体验团队与内部开发者效能团队（Coding Agents）双轨推进。
- **AI 安全防御机制**：针对非可信输入、提示词注入与私密数据外泄的“致命三要素”建立专项防线。
---
### [40:27](https://www.youtube.com/watch?v=TH6kW8KCDeQ&t=2427s) 定制化未来趋势：蒸馏小型化、开源生态与双轨治理
在圆桌的收尾讨论中，嘉宾们就模型定制化的未来趋势达成了鲜明共识：绝大多数业务场景应始终秉持“先以基座模型（Base Model）验证可行性，必要时再做定制化”的原则。Karan 提到了高并发落地中的关键矛盾：虽然 120B 等级的大模型能完美解决问题，但其推理成本极易呈指数级失控（Balloons quite a lot）。此时最佳实践是利用大模型作为教师网络生成高质量任务样本，**将能力蒸馏至 4B 左右的小型模型（Distill down to 4B Parameter Model）**，在规模化部署中实现断崖式降本。核心逻辑应当是：“先让场景跑通上线，后续再考虑成本优化”（"Make your use case work, you can worry about optimizing the cost later on"）。
Shruti 预测，开箱即用（Off-the-shelf）的通用基座模型性能将持续跃升，配合 RAG 即可满足大部分日常应用；与此同时，开源与开放权重模型（Open Weights）在未来必将占据举足轻重的地位。她以 GLM-5.2 等开放成果、Hugging Face 安全事件中开源模型展现的入侵防御能力，以及英伟达牵头、Mozilla 等多方加入的开源联盟为例，论证了开源生态的不可替代性。
最后，Edmund 勾勒了高校在系统治理上的长远双轨格局：
- **行政与合规管理侧**：全面拥抱成熟的商业前沿模型（Frontier Models），依托大厂合规机制严控运营风险；
- **科研、教学与学术探索侧**：坚定依托国家研究平台（National Research Platform, NRP）与开源权重模型，供师生自由下调和修改参数以满足深度科研需求。
这种商业前沿模型与开源开放权重并存的双轨制模式，将在未来数年内持续成为各行业兼顾风控与自主创新的主流范式。
## Notable quotes
> [04:13](https://www.youtube.com/watch?v=TH6kW8KCDeQ&t=253s) 以达成业务目标所需的最小定制量为原则（do the least amount of customization） — Edmund Clark
> [04:13](https://www.youtube.com/watch?v=TH6kW8KCDeQ&t=253s) 榨取最后 5% 到 10% 的边际收益 — Shruti Kamath
> [14:47](https://www.youtube.com/watch?v=TH6kW8KCDeQ&t=887s) 永远不要去自行研发供应商已经在做或即将推出的产品，能买现成的就买现成的，避免重复造轮子。 — Edmund Clark
> [18:22](https://www.youtube.com/watch?v=TH6kW8KCDeQ&t=1102s) 微调本身的算力成本只是冰山一角，模型在线提供服务（Model Serving）的长期推理成本才是主要的资金黑洞 — Karan Kothari
> [23:01](https://www.youtube.com/watch?v=TH6kW8KCDeQ&t=1381s) 我们深知未来将由 AI 驱动，但要赋能这个未来，必须先把数据底座搞对。 — Edmund Clark
> [32:32](https://www.youtube.com/watch?v=TH6kW8KCDeQ&t=1952s) don't try to replicate what vendors are going to do — Edmund Clark
> [40:27](https://www.youtube.com/watch?v=TH6kW8KCDeQ&t=2427s) Make your use case work, you can worry about optimizing the cost later on — Karan Kothari
> [40:27](https://www.youtube.com/watch?v=TH6kW8KCDeQ&t=2427s) 先以基座模型（Base Model）验证可行性，必要时再做定制化 — Karan Kothari
## Mentioned
Edmund Clark, James Belasco, Karan Kothari, Shruti Kamath, California State University, Vanguard, Lightning AI, Mozilla, Firefox, Amazon, Zillow, PayPal, Chewy, Cursor, Reflection, DeepSeek, Kimi, National Research Platform, CENIC, T5-Flan, YOLO, OpenAI, ChatGPT, vLLM, SGLang, Houghton Mifflin Harcourt, Macmillan, Hugging Face, NVIDIA, GLM-5.2
## Video description
Edmund Clark - Chief Information Officer, The California State University
James Belasco - Head of Data Science for Advice and Wealth Management, Vanguard
Karan Kothari - Principal Engineer, Lightning AI
Shruti Kamath - Sr Director, AI and Machine Learning, Mozilla
Generative AI is powerful out of the box, but its true enterprise value comes from tailoring models to specific organizational needs. This panel will explore the spectrum of customization strategies, from lightweight prompt engineering and retrieval-augmented generation (RAG) to full-scale fine-tuning and domain adaptation. Panelists will share real-world use cases where companies have successfully aligned frontier models with industry-specific data, compliance requirements, and customer contexts. The discussion will also cover emerging tools, platforms, and vendor approaches that make customization accessible while managing cost and complexity. Attendees will walk away with a clear understanding of how to choose the right customization path for their projects and maximize ROI from generative AI.
[Source](https://www.youtube.com/watch?v=TH6kW8KCDeQ)
Source quotes
[4:13] Edmund Clark: 以达成业务目标所需的最小定制量为原则（do the least amount of customization）
[4:13] Shruti Kamath: 榨取最后 5% 到 10% 的边际收益
[14:47] Edmund Clark: 永远不要去自行研发供应商已经在做或即将推出的产品，能买现成的就买现成的，避免重复造轮子。
[18:22] Karan Kothari: 微调本身的算力成本只是冰山一角，模型在线提供服务（Model Serving）的长期推理成本才是主要的资金黑洞
[23:01] Edmund Clark: 我们深知未来将由 AI 驱动，但要赋能这个未来，必须先把数据底座搞对。
[32:32] Edmund Clark: don't try to replicate what vendors are going to do
[40:27] Karan Kothari: Make your use case work, you can worry about optimizing the cost later on
[40:27] Karan Kothari: 先以基座模型（Base Model）验证可行性，必要时再做定制化
Source quotes
[21:02] Karan Kothari: Instead of the burden becomes how can you sort of evaluate that synthetic data to make sure it really represents your business use case... that is a simpler way to go rather than trying to generate the eval dataset from ground.
[27:06] Karan Kothari: If you just change the base model and go through the fine-tuning process again, you can significantly reduce your downstream cost of inference quite a lot.
[34:08] Karan Kothari: We don't necessarily need to upskill our workforce to be the machine learning researchers... if we can teach them on how to actually curate an highly good eval set, that can actually go a long way.
Original transcript
[0:04] Good morning everyone. I'm James Bolasco
from Vanguard, the uh head of data science
[0:10] for advice and wealth management there.
And I'll be moderating the panel here with
[0:14] uh the rest of our guests. Uh first we'd like
to go through and let everybody give a brief
[0:20] introduction and talk about why they're thinking
about customization, what's the strategic goal or
[0:26] mandate or what's gotten you kicked off on your
customization journey. Why don't we start Ed with
[0:30] you? Sure. Yes. I'm Ed Clark. I'm the CIO for
the California State University System. And if
[0:37] you haven't if you're not familiar with us, the
California State University system is the largest
[0:41] university system in the United States. Uh one
out of every 20 college graduates in the US is
[0:47] a Cal State grad. So, uh pretty big impact.
And so, when we think about customization,
[0:52] we're really thinking about impact for students,
their outcomes. How can we make our students more
[0:57] successful? they graduate on time, they
get employed, those kinds of things. So,
[1:01] we're always looking at tools that will help us
achieve that. Great. Shrui, let's go to you. Uh,
[1:10] I'm Shri Kamat. I, um, I lead the, uh, machine
learning and AI organization at Mozilla,
[1:17] uh, particularly Firefox. Uh and as maybe
many of you all are already familiar uh
[1:22] Firefox is known for its trustive trustworthiness,
openness, transparency for the open web. Uh and
[1:30] um so I've been in this AI and machine learning
field for over a decade now. Uh I started my
[1:37] career as a machine learning engineer. Uh and
I was on Amazon's first deep learning team.
[1:43] Uh then I went in uh engineering leadership
roles at various places like I was at uh Zillow,
[1:49] PayPal, Chewy uh and now I'm at Mozilla and um so
when at Mozilla and Firefox when we think about
[1:58] um customization for us there's three pillars.
So the first one is like I mentioned about
[2:05] trustworthiness and transparency. So we care
a lot about um as a result privacy as well.
[2:11] So looking at local AI ondevice models. Uh so
we're investing a lot in how we can your data
[2:18] is private. Your models are downloaded on your
device. So that means there's like data security
[2:25] and you don't have to worry about um you know
access or privacy things going over the wire.
[2:33] The second is for us model choice is really
important. uh when when I talk about model
[2:38] choice is that typically a lot of browsers do have
maybe they're locked into one ecosystem and at
[2:46] Mozilla we believe in giving users the choice
and agency over the models that they want to
[2:52] uh work and use in the browser and so as part of
that we have uh users can bring their own models
[2:58] or their combination of like open-source uh models
as well and then the third uh perspect is for us
[3:06] from a cost perspective as well, customization.
So when you think about cost uh versus quality,
[3:12] there's always trade-offs there. So we're
looking at how we can um customize for that.
[3:21] Hello. Uh my name is Karan. I'm from Lightning AI.
Uh we are a new cloud provider. So uh we have like
[3:28] customers like cursor and reflection uh using our
like GPU clusters to uh pre-train their models.
[3:34] uh at the same time like we cater to enterprises
um and help them fine-tune and deploy their
[3:40] machine learning workload on our platform.
So in terms of the fi uh like in the model
[3:46] customization area our goal is uh to help customer
be successful that means like giving them like
[3:52] enough tools and enough you know range of tools
uh we we want to like make them available for
[3:56] them like starting from like how can they do like
like parameter efficient finetuning all the way to
[4:02] um RL environments on our platform so that they
can choose the right tool for their use case and
[4:07] uh and achieve their objective. Great.
We've already heard some comments that
[4:13] get to some of the reasons of why we'd use
fine-tuning, heard some things about cost,
[4:18] some things about risk and data security. Uh,
but if you think about use cases and a business
[4:23] wants to figure out what use cases are the best to
think about fine-tuning, even consider it and the
[4:28] the trade-offs, what would be the framework? What
things would you think about when you're trying to
[4:33] identify those use cases? And I know you had
some thoughts on this. Well, you know, briefly,
[4:38] you know, the first thing uh you know, we have
first of all, there's an administrative side of,
[4:43] you know, running universities and then there's
a research and academic side and I would say
[4:48] on the administrative side for sure, you know,
one of our core tenants is do the least amount
[4:53] of customization to achieve the uh business
objective. So, you have your business objective,
[4:59] you want to deliver it. If it's not working,
whatever you're doing is not working, why is
[5:04] it not working? Right? what kinds of things
could you jump in there and customize to make
[5:08] it work? You don't want to jump into fine-tuning
right away. Um, you need to think about how much
[5:12] are you willing to spend to achieve this business
objective, right? And how much risk and compliance
[5:18] is involved, right? Those are the things that
you need to think about as you're planning out,
[5:22] you know, how far to go with with these
solutions. on the academic side, you know,
[5:27] because we have access to the national
research platform, it's a whole different
[5:32] uh kind of environment where uh the scenic uh this
California AI resource that's shared across all of
[5:40] the University of California system and the CSUs
uh our faculty and students have access to that.
[5:45] They can do they have access to all these openw
weight models, right? you have uh you know Kimmy
[5:50] and uh Deep Seek and all these Chinese models
mainly and they're they're using uh they're using
[5:56] these open- source openweight models in ways that
you often do have to fine-tune them to get the
[6:01] output you want in the research project. So that's
at least part of how I would think about Yeah,
[6:07] sure. Do you have any additional thoughts? So
I would say for um uh thinking about it there
[6:13] are two uh ways I think one is like looking at
um most of off-the-shelf models I I would say
[6:19] off-the-shelf models work fairly well with just
prompting. So if you do zeros uh oneshot prompting
[6:27] um you know you can get pretty far with it when
you do have a lot of domain data uh that you're
[6:32] sitting on. Um and at that point uh I think look I
think even with the domain level like fine-tuning
[6:39] there are I think various degrees at which you you
can think of it. I think the first is um you know
[6:44] once obviously beyond like prompting looking at
like rag where you can um you know act um you for
[6:52] instance lot of the model offtheshelf models out
there they have general knowledge. So let's say
[6:58] if you had something very particular to um you
know you could have a certain workflow like I'm
[7:05] just going to make up something like uh foo.com or
like fu information about fu in your organization
[7:11] that is just particular to your organization it's
not really available it's not general knowledge in
[7:16] that case if you have something like a rack system
then it can help retrieve that right beyond that
[7:22] um if there is um if you do have um you know use
cases where it you do need to optimize either for
[7:31] quality or for cost reasons then actually u you
know taking a model fine-tuning it through like
[7:38] different strategies uh would be at that point
really to squeeze I would say the last 10 to 5%
[7:46] um would at that point I would actually encourage
like fine-tuning um there's I think even when we
[7:51] think about customization uh it comes to you
know there's the model of customization but
[7:58] going even deeper uh there's also at the
kernel and hardware layer of customization
[8:02] as well that you can apply. So I think finally
it comes to uh the performance quality and cost
[8:08] trade-offs when you think about customization
and fine-tuning. Yeah. So that kind of brings
[8:14] us around to a couple main topics. You said the
performance, cost, quality, if you need to have
[8:19] proprietary data that's available to the model,
but then I also heard a flavor of, you know,
[8:24] foundational models are pretty good. We probably
want to test those out on the use case before we
[8:30] go the next level because there is extra effort
involved, right? And so fine-tuning is not without
[8:35] cost. And [clears throat] so then I'm interested
from all of you what are different use cases that
[8:42] you have today that you've done or are developing
where you have made the choice of fine-tuning and
[8:49] how has that turned out? Why don't we start with
you Karan? Yeah. So uh I can I can talk about one
[8:56] of our customer who is in healthcare space. So uh
they have like pabytes of uh clinical reasoning
[9:02] data that is like proprietary to them. So and
uh their objective is and and they have like
[9:08] um uh the samples from like clinicians around the
world around like how they go about uh reasoning
[9:14] about you know a clinic uh clinical conclusion. So
uh they have all of that data but like if they use
[9:19] any frontier model or even like any you know large
uh open weight model uh the accuracy is like quite
[9:26] low or or uh model is not able to just reason
through their decision quite well. So given that
[9:32] they have all this proprietary data so they have
fine-tuned so that it can be like really good at
[9:37] domain specific reasoning and with that the model
that they were able to build with the help of you
[9:44] know rag because they still have to use rag to
you know um to collect the evidence and put that
[9:49] in front of the model but then they fine-tune
in a way that it can use the information that
[9:55] is in front of the model so that it can reason
through the clinical conclusion really well.
[9:59] So now that model is being used as a companion
uh for the doctors where they can use that as a
[10:06] uh as a second person that they can you know just
take an advice from. And so that sounds like an
[10:12] interesting use case. It sounds like you had to
go pretty far in fine-tuning. You had to use rag
[10:16] with the data as well as fine-tuning the model.
And you're talking about large sets of data,
[10:21] right? And so I imagine that that's uh a pretty
significant commitment by them in order to get
[10:27] the higher level of performance. Do you think
that that is completely necessary for their
[10:34] use case or like what drove them to really spend
the cost to do that? Yep. Uh so it's it's due to
[10:40] the nature of the problem. um given that this
kind of like proprietary data is not available
[10:46] for like frontier labs or other people to consume
the models themselves they are not as useful the
[10:52] way they are today. So if you want to like really
build a product that you think clinicians will use
[10:56] that investment is necessary. So of course like
before starting the investment you won't know if
[11:02] you'll be getting the ROI on the investment but
uh one has to sort of you know take the leap and
[11:07] and have conviction okay this is something that we
want to like spend uh time and effort on and and
[11:12] and it's also over the time even the base models
have improved a lot in the reasoning capability.
[11:18] So even if let's say you don't get the promising
promising results today uh but in in few weeks
[11:25] few months there may be another model that you can
try to apply like newer finetuning techniques and
[11:30] hopefully get to the objective that that you're
looking for. So it's it's more like an iterative
[11:34] process than you know we'll just do fine-tuning
once if it works we won't do anything or um we'll
[11:40] just continue with with that. Well, we've got some
questions about the maintenance cost of all that
[11:45] and having that continue uh a little bit later,
but let's go to some more use cases. Shruthy,
[11:49] do you have some examples you'd like to share?
So, we have an interesting use case called as
[11:54] smart app grouping. Uh so, in Firefox, it will
group related tabs together and also suggest
[12:00] uh group name as well. And so, this is entirely
powered by an ondevice model. And uh for the group
[12:07] name suggestion, we actually fine-tuned a model uh
for there were a few reasons for this one was that
[12:13] u most of the models that we had u when it came
to giving a group name suggestions they were too
[12:18] general and what we found that u why someone might
group a tab could be very context driven. Uh so it
[12:27] could be because maybe you know this is a trip
that you're planning with a friend or this could
[12:32] be related to a um activity that you're trying
to do and so not every time you know it's the
[12:40] names that we were giving through these models
were very like general like it's travel related
[12:45] and we wanted it to be much more sort of specific
and the other reason for fine-tuning was that we
[12:50] wanted to make sure this model worked on as many
different types of devices which had different
[12:55] hardware configurations like memory requirements
and CPUs and and GPUs and so what we found was
[13:03] that fine-tuning actually we wanted a really
small model so they can cater to a large set of uh
[13:08] devices as well uh and so we ended up um sort of
fine-tuning a very uh small model which was the T5
[13:17] flat although this is a relatively old model but
it worked very well in this use case and the model
[13:23] size was less than like 60 MB Uh and we saw as
well in both in terms of uh quality as well as in
[13:31] in the performance we saw a huge board. So it was
like more than doubled um than you know just using
[13:37] something off the shelf. Yes. So you brought some
good additional considerations here like that's
[13:42] very clearly a case of where you need a smaller
model from the infrastructure side. The hardware
[13:47] that it's running on has to host the model itself.
Therefore, you have to get a smaller one that can
[13:54] push you directly in that direction. And as well,
you're still getting good performance out of the
[13:59] smaller model. Did you test big model versus small
model and look at performance? And you said that
[14:04] you looked at the general case, but did you look
at bigger models before the smaller model as
[14:08] well? We did look at bigger models, but I think
for us the main thing was that bigger models,
[14:15] the devices that it could run on would be not that
much. And then where we use bigger models is more
[14:20] for the labeling side of things. And so really
generating the label so that then the small models
[14:25] can actually uh we can use that for fine-tuning.
So that's how we mainly used the bigger ones. And
[14:30] did you get a big performance trade-off or was
the small model just as good? Uh no with the
[14:35] with sort of the fine-tuning and everything we
could get there was um there was a very little
[14:40] degradation but it was like in terms of like you
know this perceived quality it was almost at par.
[14:47] Nice. Nice. All right, Ed. Yeah, I'll talk about
um a failure, a customization, and there was a
[14:54] particular reason it failed, but and then also,
you know, a fine-tuning case. But briefly, um you
[15:01] know, we've already heard from our panelists here,
you know, again, these models that are already out
[15:06] there are very good and hopefully it'll solve your
business problem. Um, and in fact, uh, what we're
[15:12] finding in our Cal State system is as these models
get more and more advanced, there's there's like a
[15:17] gap between what the users are using these models
for and what's what they're actually capable of,
[15:22] right? And that's growing uh, as time goes on.
So 95% of what we want to do can be done simply
[15:29] by using the base model using some rag and maybe
you know the skill have a skills file and you can
[15:36] probably accomplish most of what you want via
that mechanism. One of the things that we were
[15:42] very excited about because you know Cal State is
a large system. Many students have affordability
[15:47] issues you know like a first family pell grant
recipients and so we were trying to promote open
[15:54] textbooks and this gets to another principle like
one of the principles we try to adhere to is don't
[15:59] try to build something that a vendor is going to
build because then you can buy it off the shelf.
[16:03] You don't want to replicate all that work and
then say oh now this company's built it and we
[16:08] should just buy that instead. So we didn't want
to invest but in terms of open textbooks which
[16:13] could save our students hundreds of millions
of dollars. Textbooks are very expensive
[16:18] um and trying to adopt textbooks open textbooks
is difficult for faculty members because if you
[16:24] buy that you know you know that Mifflin textbook
it's going to come with all the teacher resources
[16:30] quizzes testing and all that stuff. So
trying to build a uh open textbook tutor
[16:37] um that speaks in all the languages that you
know could actually contain all the content our
[16:42] students would need in different subject areas was
a project we launched and getting to the cost what
[16:47] happened is it you know faculty members would say
I want to use this textbook and I don't want it to
[16:53] go outside this textbook. What we learned is when
you only have a very small amount of material it
[16:57] wasn't really enough to have the outcomes that we
wanted. So we had to add more textbooks. That was
[17:02] one problem. All these other things that had to
happen. But we kept customizing this thing and
[17:07] started to see that the cost to make it really
sing, you know, putting 200 textbooks in it,
[17:13] controlling how it did the tutoring so that
it wasn't just giving answers, but actually
[17:17] like providing that cognitive dissonance so that
students were actually learning through this. It
[17:22] ended up being so expensive that we were trying
to get other universities to join in. We were not
[17:27] successful in that. I still think it's an amazing
idea but would require a consortium to build this
[17:32] and it would be hugely expensive otherwise. So
um and then you know a fine just you know a good
[17:39] example you know one of our faculty members
is really into smart manufacturing and uh and
[17:44] one of the the open- source uh models he uses is
called YOLO. It's you only look once it's an image
[17:50] detection thing. Some of you look like you know
what we're talking about. and that um that YOLO
[17:56] model in you know it doesn't know out of the box
what your factory floor looks like. So you will
[18:02] have to say hey this is what it's going to look
like in this environment and so you're going to
[18:06] have to provide different you know different kinds
of feedback to make it work there. Um so that's uh
[18:12] the other thing the output for his research needed
to come out a certain way not out of the box way
[18:17] but the way that he needed it for his research
paper to make sure the parameters were the same.
[18:22] Yeah. Very interesting. Definitely different use
cases showing up across the board here, but that
[18:27] seems to speak to the variety of the options that
are available with this. Uh to the cost point,
[18:33] you're teeing us up to the next question here. You
know, fine-tuning it can become expensive. Uh you
[18:40] can have to use a lot of compute in order to do
it, especially if you're trying to do something
[18:44] complicated and there's lots of platforms to help
do it. We've got uh lightning AI here. So uh Karan
[18:51] how how do you think about this like the breath
of data required platform requirements how do you
[18:58] get the fine-tuning and the performance while
minimizing the cost yeah uh so if we like look
[19:05] at fine-tuning we cannot just look at the cost of
the finetuning because I think that is like part
[19:10] of the cost the the primary cost they comes from
actually serving the model itself and especially
[19:16] if you are actually doing serving at a um large
[snorts] scale because your cost can actually add
[19:22] up quite a lot. So I think if you will like look
at the decision framework on like how should one
[19:27] sort of approach that like there are like three
knobs that one has to think about like what is
[19:31] the quality that you're okay to live with and then
how much cost are you willing to pay and what kind
[19:37] of control that you need uh in when it comes to
like fine-tuning at the same time model serving
[19:42] itself and and assume that okay there like let's
say there are like multiple models or so so even
[19:49] when we start fine- tuning we have to figure out
okay what is the base model that we're going to
[19:52] use to fine tune there like whole sets of like
whole range of candidates that we can pick from
[19:58] but if we just like pick each and everyone and
then try to fine-tune uh the the cost will be
[20:03] quite high so I think in order to like narrow
down the list we would recommend first look at
[20:08] how does your serving platform looks like does
that platform have any constraint in terms of
[20:12] the architectures that it can support or do you
have like any SLAs that you're targeting in terms
[20:18] of the throughput and the latency and that can
help you like narrow down the list of candidates
[20:23] model quite a lot. And once that is done, having
a really strong eval set is extremely important
[20:30] because if let's say if your eval sets are
changing through uh throughout the process,
[20:35] then essentially you're just paying the cost
of fine-tuning again and again because you have
[20:39] to like keep on uh redoing the eval. So having I
think having the last mile locked down and having
[20:45] the first mile locked down as early as possible,
it can really help reduce the cost associated with
[20:51] the finetuning in general. And you you talked a
lot about model choice and then infrastructure,
[20:57] but then you got to the point about what's the
evaluation data. There's a data requirement here,
[21:02] right? have to have the appropriate information
in order to do the fine-tuning and that can
[21:09] sometimes be hard to develop itself.
Right? So I'm interested, you know,
[21:14] we talked about a couple use cases here. What have
been the strategies in order to even get that data
[21:19] to do the fine-tuning because that can be part
of the challenge for the total system itself.
[21:26] So uh one of the thing is of course like you can
use your own proprietary data or let's say if you
[21:31] know what is a successful business outcome
looks like that is definitely a way to start
[21:36] uh but then it can become extremely like
timeconuming and tedious to actually generate
[21:41] uh enough number of samples that can that are
required to like do go through the finetuning
[21:46] process. I think that's where like some of the
frontier models they they really like come to
[21:49] your rescue because like once you give them offer
with you know some of the high quality examples
[21:54] you can like synthesize rest of the data set and
then you can play with that and and I think that's
[21:59] where then instead of the burden becomes how
can you sort of evaluate that synthetic data
[22:04] to make sure it really represents your business
use case but I think that is a in my opinion
[22:09] that is a simpler way to go rather than trying
to generate the eval data set like from ground
[22:16] So synthetic data comes to play true. You want
to add I wanted to say since in Mozilla we don't
[22:21] uh we care about privacy. We don't use user data
and so synthetic data is one of the ways and using
[22:26] some of these frontier models for just like that
label generation and we use also like human evals
[22:32] um and then we then apply data augmentation
and so then you can get to the scale of what
[22:37] you need for like the fine-tuning. So in the
example that I mentioned for with smart tab
[22:42] grouping we did exactly that. It was a lot of
synthetic exa data which was um wetted with
[22:48] like human labels and then we sort of scaled that
to re to meet the data volume needed for funing.
[22:57] I the only thing I would add is you know uh you
know and we have a brand new chancellor and she
[23:03] had some very bold goals like you know she wanted
the graduation rate to increase by this amount and
[23:08] the employment outcomes to increase and you know
all these kinds of very bold goals that were so
[23:14] clearly delineated that actually in you know my
system which includes 22 different universities
[23:20] and 22 presidents and 22 CIOS and everything else
that kind of bold goal actually forced us to come
[23:28] together because everybody's used to operating
very in very siloed pockets right doing their own
[23:33] way defining majors their own way all that kind
of it was such a mess and then having these goals
[23:39] actually got us together and like we know this is
going to be an AI enabled future but to enable it
[23:44] we have to get this data thing right so going to
this unified data platform and approach sort of a
[23:50] cookbook of how we're going to approach this data
problem these are the technologies we're going to
[23:54] use that's helping us build the foundations to be
able to approach some larger problems in the CSU.
[24:03] [clears throat] I think when you spoke about cost,
I want to add like a few more things and I think
[24:07] a lot of the costs with LLMs are generally at
inference time. Obviously there is training but
[24:11] inference is where um I think it can get expensive
quite quickly. Uh so I think this is more from the
[24:18] point of view of like customization. So I would
think like there are few layers of that. one is
[24:23] at the model layer, right? Uh for optimizations
there, you can apply things like quantization,
[24:28] distillation. Um where uh you can effectively
maybe bring the model to a much smaller size.
[24:36] Um so that that's I think one of the strategies.
Uh the second is like from beyond the model itself
[24:43] there is the serving layer where you can uh have
a lot of like caching. Um the speculative de uh
[24:49] decoding is one of the other ways um where you
can um you know it generally there's a small and
[24:55] a larger model uh with speculative decoding and so
there um most of the time you're using the smaller
[25:00] model. So that's one of the ways um and then
going even like deeper in the stack there is at
[25:06] the kernel layers as well you can you know bring
custom kernels um that you can apply uh that are
[25:12] more customized to your data and your use case. uh
and then also thinking about the right choice of
[25:16] the hardware and GPUs that will uh bring that
sort of cost optimization. So I think this is
[25:22] how I would sort of frame thinking about cost
especially at inference which is where I would
[25:27] say majority of the people definitely you know you
might not be training or fine-tuning but you're
[25:33] definitely thinking applying inference. Yeah. So
um one thing to add on to just on the quantization
[25:40] related aspects like given that using those kind
of technique like there could be like minimal
[25:45] uh quality in the loss but I think that's where
like uh picking the right candidate becomes very
[25:51] important like if you don't have the wiggle room
in terms of the you know degradation in quality it
[25:56] there just like reduces the option that you have
down the line to uh make this optimization. So I
[26:02] think that's uh one of the key thing at least we
have noticed with operating with few customers
[26:09] you have to add I probably have less to add
to this particular question than most partly
[26:15] because you know we have a systemwide chat GBT
deal and basically our environment is you know
[26:24] has been pretty much go do whatever you want right
and now that now that openai is saying hey we're
[26:30] going to charge you for the credits you're using
that's suddenly causing us to do a big kind of
[26:34] reconsideration. wait what is this going to mean
for all of us and our usage and we [snorts] can
[26:38] see this person went way over their limit and
that's a new so it's a new thing for us to start
[26:44] considering and similarly on the national research
platform the NSF is funding it we have grants from
[26:48] all these faculty so they're [snorts] just using
it and in fact uh um you know we have a thousand
[26:53] GPUs in the California setup where you know these
researchers can do a lot of work on it and they're
[26:58] not getting a massive bill from it right so we're
newer to this than you all are. So that brings
[27:06] up a bunch of questions about now total cost of
ownership. We have these use cases. We've decided
[27:13] that fine-tuning is a good choice for them. Some
of them may be newer than others, but then you get
[27:17] into this cycle as you alluded to Karan about
they have to be updated. You're now owning the
[27:22] fine-tuned solution. Some of the parameters around
that might change over time. So then you have to,
[27:28] you know, pay for the infrastructure to
handle that. You might have to host your own
[27:31] infrastructure. you might have to host your own
hardware in order to do that and then you have to
[27:36] maintain and update these things as things change
which is uh an additional part of the calculus for
[27:42] the use cases. So how have you all thought about
total cost of ownership and how that uh impacts
[27:48] whether or not someone should do fine-tuning?
Yeah. So I think we would actually recommend
[27:54] um you know we recommend customers to actually
keep fine-tuning their models as the time goes on
[28:00] because if you like change the base model itself
uh because what we are seeing like every let's say
[28:05] every few months there's a new base model that
emerges u and these are like open weight models
[28:11] for half of the size the there is no like there's
marginal degradation in the intelligence that it
[28:16] offers. So given that uh most of the fine-tuning
real costs they come at the inference time even
[28:22] if you don't change your like eval like data
that you use for finetuning a lot um and if
[28:28] you just like um just change the base model and
just go through the finetuning process again you
[28:33] can significantly reduce your downstream cost of
inference quite a lot. So I think instead of like
[28:39] sticking with you know what's working I think it's
I think it's important just to keep experimenting
[28:43] with the new strategies that are coming in and
new ways of actually serving the model with like
[28:48] new strategies and and especially like if you
actually try to use the latest and greatest
[28:52] base model a lot of this VLM SGLang this serving
containers they come with you know some native
[28:57] optimizations that can significantly like uh
increase the throughput at the time of inference.
[29:03] So, it sounds like you're you're advocating that
there's significant value, new base models come
[29:08] out, new things that you can fine-tune, you should
probably have part of your process be uh refinet,
[29:15] shall we say, on the on the new model with the
existing set of evaluation data. Does that sound
[29:20] right? So, I would say like when it comes to like
total cost of ownership, you're looking at it from
[29:25] like um from the perspective of what it takes to
build, what it takes to actually run these models,
[29:31] what it takes to maintain, right? and any other
probably like compliance that you know sort
[29:36] of looking at it like holistically and I think
it also depends on your use case as well right
[29:42] um I think whether this is a net new use case
net new is in the sense it's not yet proved out
[29:48] it's not yet in the market versus if this is
something that is incremental where you have
[29:52] probably more data to build off of to then decide
you know what is going to be your projection of
[29:58] cost let's say in the next 6 months to one year
to like maybe you know down the line for larger
[30:04] number of years as well. So, uh the way I would
say that um few things and like Mozilla AI is
[30:12] been fairly new and so we've been looking at a
lot of like net new use cases. Uh so one of the
[30:17] things that we recently introduced was like the
smart window which is a new AI assistant into
[30:22] the browser and so when we think about like cost
obviously there's a tons of like customization and
[30:27] maybe to a certain degree even fine-tuning but you
know one of the things that I want to mention is
[30:32] uh which maybe there's you know a lot of people
could be in the same boat is like when you think
[30:37] about cost you're also think about like uh vendor
lock in as well and how much do you want to plan
[30:43] for the future. So I would say that if you're
relatively new and you are still figuring out
[30:48] like let's say product market fit then you want
to be very conscious about not trying to have a
[30:53] particular vendor lock in and to really have
that optionality. Uh so um and so one of the
[31:00] things that we did was that really went with like
pay-per-use models and vendors as well. So it's
[31:06] rather than trying to buy GPUs with which are
generally much cheaper if you went with like a
[31:12] three-year or a 5year commit. And so we were like
okay very consciously uh it's okay if the spend in
[31:19] the near term is a little high but it helps us vet
the use case determine scale determine what volume
[31:25] we might want to fine-tune and and then we can
invest more into um actually projecting better
[31:32] costs so that we then we can you know determine
our total cost of ownership um uh which is which
[31:39] includes like infra to other maintenance to even
like human cost. So that's how we've approached it
[31:44] in our case. I got it. So you're proving out the
use case uh being appropriately agile as you start
[31:50] for doing that and that helps you understand
better what your total cost would be and then
[31:54] I guess there's an evaluation decision sometime
later is whether or not that that's worth it. Ed,
[31:59] did you have more to add? Well, you know, and and
u this you know, I mentioned the uh CSU student
[32:05] companion, our our effort that is now, you know,
on the on the ice or whatever you would call it,
[32:10] but um and you know, obviously uh the reason why
we went down this path is you know, these large
[32:17] publishers like H Hotton, Mifflin, and McMillan
and so forth, they're going to put out textbooks
[32:21] and they are starting to add this sort of like AI
tutoring component to their textbooks as well for
[32:27] an extra charge. But uh you know I teach in the
business school at Fullerton and you know the
[32:32] two books that I have for my class. Each of them
is about $400. That's a big deal for students,
[32:37] right? And so I think the the idea that we could
actually do something for our students giving them
[32:44] free te textbooks with the tutoring and all these
other resources is a compelling uh pitch. But the
[32:50] problem is it's very hard for the faculty to jump
in without everything being there. So part of this
[32:56] cost of ownership and you know like the sort of
life cycle is like you're still piloting stuff
[33:00] and they need something complete cuz they're
teaching in class right now. And I think that
[33:05] the gap between us getting there and the timing I
I just think uh I think it's it gets to the point
[33:12] uh don't try to replicate what vendors are going
to do. One of the biggest problems we have right
[33:17] now is a lot of transcripts come on via paper and
everybody says oh we should have an AI that scans
[33:22] the transcripts and you know puts in the correct
grades and all that stuff and we're like well you
[33:26] can do that but of course I assume every document
management company in the world is also trying to
[33:31] do that right now too. So don't necessarily want
to invest in that unless you can find a really
[33:37] big you know savings and you can manage it to
your point you can manage all the updates and
[33:42] upgrades and everything else. I think these
are all very wise points that my colleagues
[33:47] have made and I think you've got some interesting
considerations versus technology companies where
[33:53] maybe it's much more native for them to own some
of this infrastructure and just own these systems
[33:59] outright or keep them in-house versus go with a
vendor solution. There's plenty of build versus
[34:04] buy talks here. We don't have to necessarily get
into that directly. Uh but why don't we pivot a
[34:08] little bit like we talked about total cost of
ownership and like what the use cases are but
[34:14] there's a whole part that is the people and team
development part right you know everybody sees the
[34:20] latest and greatest foundation models you may have
ability across your organization to have different
[34:26] teams just use those out of the box they may not
be incentivized or even understand the benefits of
[34:32] customization and fine-tuning for their specific
use cases but You want to be able to federate this
[34:39] type of capability across your organization
so that you can get benefits across multiple
[34:43] different areas not just you know one use case
at a time ideally. So how do you guys think about
[34:51] enabling fine-tuning and customization across an
organization as part of the capability set? How
[34:57] do you upskill folks that are uh part of your
organizations in order to make them understand
[35:03] that that's available and be thinking about those
costbenefit trade-offs? Ready to go K? Yeah. So I
[35:09] think uh one thing that I will add to that is if
we just look at the finetuning itself like we we
[35:16] don't necessarily like need to like uh upskill our
workforce to be the machine learning researchers.
[35:22] Um I think it's it's the so the first part
is if we can teach them on how to actually
[35:29] curate an highly like good eval set that can
actually go a long way because then that can
[35:34] allow you to figure out whether you should even
use model customization or not or should you just
[35:38] use like normal fine tuning or like prompt optim
optimization techniques or should you just do uh
[35:44] uh or or or just rag will work in that particular
situation. That's why I think that part is most
[35:51] critical if I were to like recommend where should
you know people spend effort on because I think
[35:55] the downstream things on how to fine-tune and
how to run the models I think there's there like
[36:00] enough infrastructure that is out there that that
just makes that uh part quite easy these days but
[36:05] just generating the highly you know high quality
eval said that is that represents the business use
[36:11] case I think that's the that's the most difficult
part and I think that's where organizations
[36:16] they can um they can upskill of their workforce.
Interesting truth. So I think uh centralization of
[36:26] this would make a lot of sense in an organization.
When I talk when I say centralization, I really
[36:31] thinking about having uh leadership sponsors as
well who are supporting uh because AI is anyway
[36:37] is becoming part of everyone's daily lives very
quickly and it's moving changing at a fast rate
[36:42] as well. uh so building that your organization
thinking you know how to incorporate that I think
[36:48] that's definitely I would say um uh sort of the
way to go and uh when you're thinking about making
[36:56] this work in an organization having some central
uh uh centralization would help which means that
[37:02] there are few dedicated people thinking about
um bringing like the latest technology how do we
[37:08] uh where are where is people at in the AI
understanding in the organization and what
[37:12] are steps that we need to take to kind of
uplevel them. So some of the things that
[37:18] um we've done at Mozilla is that like my org is
focused a lot on product use cases and building
[37:25] you know uh for consumers out there and users of
the product. There's another org thinking about
[37:30] AI internal developer productivity and bringing
in like coding agents and how do we make that
[37:36] very efficient um for like de developers internal
workflows. So I think that level uh of investment
[37:43] can go a really long way uh because even to get
these tools there's a lot of maybe internal like
[37:49] enterprise you need to work with different like
your legal and policy teams on this um as well
[37:54] as setting the right guidelines for how to use it.
Um and I would the other thing that I've se worked
[38:01] well in my organization that I've done is building
like champions for uh AI in the organization who
[38:08] work at different problem spaces. So uh you could
have someone thinking about specifically like the
[38:14] latest AI models and how it works across all use
cases and you know having reports on that for
[38:20] um you know everyone in the organization to
consume. Uh uh recently we had uh one of my
[38:27] uh senior engineers look at how to h uh what
are the strategies for uh AI security related
[38:34] use cases. So um there's something called as
the lethal trifecta which is that if you have
[38:40] um untrusted content uh prompt injections are
very common uh you can have your private data
[38:46] set AI can send that to um you know can send it
out to some evil agent and so we don't want that
[38:53] to happen and so what are ways in which we can
mitigate that and so one of these engineers um
[38:58] looked at strategies to mitigate it came up with a
plan and then um you know we got that done and so
[39:05] this is such an important step because there is
room for that uh looking at you know what's the
[39:11] latest and greatest out there and bringing those
strategies. So I think it's also about building
[39:15] that culture of like innovation where there's
room for people to come in and suggest propose new
[39:22] ideas and and um sort of upskill and uplevel so
that everyone else can adopt that. So yep. Yeah, I
[39:30] think there's definitely a cultural component and
then and I think we've seen ourselves that when
[39:36] you tie things to the product space where you need
to deliver this and it has specific constraints,
[39:42] you have to look at the options that go with
those constraints. And we've already talked about
[39:46] a bunch of those options as being the driving
factors for why you'd even choose fine-tuning.
[39:50] I think a key point I heard from you Karan was
that the evaluation data sets I think there's
[39:57] probably just business users non-technical folks
who are key folks as part of that and as they work
[40:03] with the technical [clears throat] teams in order
to deploy solutions like there's significant value
[40:08] add from that side and they can help in to help
the technical teams understand the depth of the
[40:17] process that they're trying to update or or what
from the business side needs to go into the AI
[40:22] solution, but those can be tricky to develop.
Okay, we we've only got a couple minutes left,
[40:27] so uh just a tossup for everybody, you know,
fine-tuning, customization, new big open models,
[40:36] uh or open weight models are around these
days. So, uh begs the question, what do
[40:43] you all think about the future of customization?
Are more things customized? there more things on
[40:48] uh open weights fine-tuned. Do you see that
as like the higher value for most of your use
[40:53] cases or do you think there's still just a large
space that's just we're going to take the base
[40:58] model that's available and it's good enough?
What do you think? Why don't we start? Yeah,
[41:02] I think in most cases it's always I think better
to start with the base model and dive into the
[41:09] customization unless like it is absolutely
necessary. So one of the places where we have
[41:14] seen people really have to do that is let's say
even if the base model is working perfectly fine
[41:18] but they're having to use let's say 120 billion
parameter model to to run their use case. So the
[41:24] cost actually balloons quite a lot. So can we
and and if the use case is just specific to one
[41:29] particular task can we just you know generate
like high quality example and just distill it
[41:34] down to like 4 billion parameter model and just
uh reduce the cost at scale. Uh but I think the
[41:39] first thing is just make your use case work.
You can worry about optimizing the cost later
[41:44] on. Great truth. I I would say that um uh most
of the off-the-shelf models I think are going
[41:54] to continue to get really much much better and so
that will I think work for majority of the people
[42:00] uh use cases and maybe in combination with
rag I think that's the way to go. Uh I think
[42:04] um again I think open-source models are becoming
more and more uh capable as well as we've seen
[42:11] with like the GLM 5.2 uh I uh and with the
recent like hugging face incident as well. We
[42:17] saw the open source model was able to like c the
um the intrusion as well and to and so there's the
[42:23] open source alliance as well that Nvidia started
and many players including Mozilla are on that.
[42:28] So I I I think that um um open source is also
going to and open worlds we models are going
[42:35] to have a long um you know I think they are
also going to be a big part of the future.
[42:42] Yeah, I would just add that I I think the division
we're seeing today on the administrative side,
[42:47] we are definitely using the frontier models
and all you know like this is what we feel
[42:51] safe in this you know we have we don't want to
take all the risk on the teaching and learning
[42:55] and research side I think our students are
going to continue and our faculty are going
[42:59] to continue using things like the national
research platform using open weight where
[43:03] they can come down and change the parameters
as necessary to support research. I think
[43:08] that that division will still exist for many
years to come. All right, great. And that is
[43:14] pretty much the time that we have. So thank you
everyone for a good discussion and on to the next
