Cox Automotive Caps AI Costs by Routing Work Away From Frontier Models
Ai4
Defaulting to massive foundation models for simple enterprise tasks triggers severe budget variance and latency, making automated query routing and calibrated evaluations essential to viable AI deployment.
## Background
Published on YouTube by the Ai4 channel on August 14, 2026, this presentation features Cox Automotive's machine learning leadership detailing architectural frameworks for enterprise AI. The talk addresses the explosion of model architectures—from frontier LLMs to fine-tuned SLMs and edge inference—and explains how teams can balance speed, safety, and cost without defaulting to whatever model is most familiar.
## TL;DR
Cox Automotive Director of Machine Learning Udit Luther outlines the three primary traps of enterprise AI: unpredictable cost variance, confident hallucinations, and high latency that causes users to abandon workflows. Instead of treating massive foundation models as a one-size-fits-all hammer, Luther presents a structured toolbox incorporating deterministic APIs, automated query routing, fine-tuned Small Language Models, and edge deployment. The session details how to calibrate and pin LLM-as-a-judge evaluation pipelines while avoiding common benchmarking pitfalls. Finally, Luther introduces four strict decision gates and highlights the hidden operational liabilities organizations must evaluate before committing to custom model training.
## Atmosphere & style
The presentation carries a grounded, candid, and highly pragmatic tone, stripping away generative AI hype in favor of hard-learned production realities. Luther communicates with relatable, self-deprecating humor and sharp analogies, comparing looping agents to teenagers handed corporate credit cards and illustrating latency stakes through roadside fleet mechanics working next to 70 mph traffic. It feels like an honest architectural post-mortem and field guide, balancing corporate financial discipline with practical engineering rigor.
## Key takeaways
- [00:04](https://www.youtube.com/watch?v=yOQUFKJ0x6w&t=4s) Enterprise AI deployments face three critical operational bottlenecks: runaway cost variance, confident hallucinations, and creeping latency that degrades user experience.
- [03:02](https://www.youtube.com/watch?v=yOQUFKJ0x6w&t=182s) Human difficulty diverges sharply from model difficulty: LLMs effortlessly handle complex calculations like depreciation curves, yet easily fail at basic common sense like calendar math or vehicle trim specs.
- [06:13](https://www.youtube.com/watch?v=yOQUFKJ0x6w&t=373s) Latency is the primary cause of user abandonment in field applications, meaning teams must replace brute-force LLM inference with fast, deterministic code and APIs for basic lookups.
- [11:04](https://www.youtube.com/watch?v=yOQUFKJ0x6w&t=664s) High-efficiency AI architectures rely on intentional routing, placing lightweight intent classifiers at the front to dispatch simple prompts to cheap models and reserve frontier models for complex reasoning.
- [16:10](https://www.youtube.com/watch?v=yOQUFKJ0x6w&t=970s) Evaluation benchmarks frequently fail due to four traps: multiple-choice recognition bias, self-recognition bias on synthetic text, prompt phrasing mismatch, and contaminated human ground-truth data.
- [19:51](https://www.youtube.com/watch?v=yOQUFKJ0x6w&t=1191s) LLM-as-a-judge pipelines must be rigorously calibrated against expert human gold standards and version-pinned to prevent evaluator drift across experiments.
- [22:44](https://www.youtube.com/watch?v=yOQUFKJ0x6w&t=1364s) Fine-tuned Small Language Models (SLMs) regularly outperform generalist frontier LLMs on domain tasks (85-95% vs. 70-85% accuracy) at approximately one-tenth of the inference cost.
- [25:49](https://www.youtube.com/watch?v=yOQUFKJ0x6w&t=1549s) Model serving infrastructure dictates financial risk, ranging from zero-idle-cost on-demand APIs for bursty traffic to high-fixed-cost dedicated provisioned endpoints.
- [28:55](https://www.youtube.com/watch?v=yOQUFKJ0x6w&t=1735s) Cost optimization requires shifting from passive downstream accounting to proactive upstream controls via strict governance boundaries and automated query routing.
- [31:57](https://www.youtube.com/watch?v=yOQUFKJ0x6w&t=1917s) Organizations should pass four sequential gates before fine-tuning: having proprietary data, robust benchmarks, exhausting simpler prompt/RAG methods, and verifying sustainable hosting economics.
- [35:56](https://www.youtube.com/watch?v=yOQUFKJ0x6w&t=2156s) Custom model training introduces heavy hidden liabilities, including unpatchable data corruption that requires full re-training and internal ownership of safety alignment.
- [38:59](https://www.youtube.com/watch?v=yOQUFKJ0x6w&t=2339s) Individual foundation models quickly become obsolete, but the evaluation benchmarks, guardrails, and architectural routing frameworks built around them remain durable enterprise assets.
## Detailed walkthrough
### [00:04](https://www.youtube.com/watch?v=yOQUFKJ0x6w&t=4s) The Three Core Headaches of Enterprise AI
Udit Luther introduces the session on behalf of himself and his leader, Jodi Blomberg (VP of AI & ML at Cox Automotive), who could not attend. Luther leads Generative AI evaluations and model management across Cox Automotive, framing the discussion around three acute pain points engineering and data science leaders face today. The first is bill shock: engineering teams opening cloud invoices and watching their eyes pop out at runaway expenses. The second is confident hallucination: models being "beautifully wrong, but still so confident" that incorrect answers easily slip past scrutiny. The third is creeping latency: users growing impatient when waiting ten seconds for an interactive turn.
```mermaid
flowchart TD
accTitle: The Three Core Enterprise AI Challenges
subgraph CoreProblems["The 3 Enterprise AI Traps"]
P1["1. Runaway Cost & Variance<br/>Inference, loops, hosting, evals"]
P2["2. Confident Hallucination<br/>Plausible yet factually flawed"]
P3["3. High Latency<br/>Multi-second delays killing UX"]
end
subgraph Solutions["Engineered Solutions"]
S1["Agentic Design & Tooling"]
S2["Calibrated Evaluations"]
S3["Model Routing & Sizing"]
end
P1 --> S1
P2 --> S2
P3 --> S3
```
- Autonomous AI agents exacerbate cost because they are goal-oriented at any price; they will loop repeatedly, re-read full context windows, and invoke irrelevant tools until they reach an exit condition.
- Total cost of ownership goes far beyond base token inference, incorporating dedicated infrastructure hosting, observability pipelines, real-time security, and evaluation harnesses.
---
[03:02](https://www.youtube.com/watch?v=yOQUFKJ0x6w&t=182s) The Teenager with a Credit Card: Cost Variance and the LLM Paradox
Comparing autonomous agents to teenagers given corporate credit cards, Luther notes that the real anxiety for finance leaders is extreme budget variance: bills fluctuating wildly from $40,000 to $4,000,000 without predictability. The slight saving grace over teenagers is that agents usually complete their assigned tasks. Compounding this challenge is the divergence between human difficulty and LLM difficulty. Tasks that are intellectually rigorous for human experts—such as calculating complex vehicle depreciation curves or drafting dense essays—are trivial for frontier models. Conversely, basic common-sense facts trip them up: an LLM might effortlessly calculate complex math while hallucinating a V8 engine inside a four-cylinder Toyota Camry.
```mermaid
flowchart LR
accTitle: Human vs Model Difficulty Disconnect
subgraph EasyForHuman["Easy for Humans / Hard for LLMs"]
H1["Calendar math & appointment dates"]
H2["Basic vehicle trim configurations"]
end
subgraph HardForHuman["Hard for Humans / Easy for LLMs"]
M1["Calculating complex depreciation curves"]
M2["Generating long-form structured essays"]
end
```
- In early agent deployments for dealership service booking, models confidently scheduled customer appointments for "Friday, August 5th" when August 5th fell on a different day of the week.
- Defining ground truth in open-ended generative domains is inherently ambiguous; recommending a Porsche in response to a query for a "family-friendly car" is subjective and difficult to evaluate automatically.
---
[06:13](https://www.youtube.com/watch?v=yOQUFKJ0x6w&t=373s) Roadside Latency and the Law of the Instrument
Latency is routinely treated as a post-deployment optimization, yet it is the primary reason end users abandon software. To illustrate, Luther highlights Cox Automotive's fleet services technicians standing on highway shoulders with 70 mph traffic speeding past. A technician waiting for an automated repair estimate and cost approval needs an immediate answer so they can turn wrenches and safely leave the roadside. If an agentic system requires ten-second conversational round-trips to deliberate, the technician will simply abandon the application and place a manual phone call.
The root cause of these systemic issues is the classic law of the instrument: "if all that you have is a hammer, everything looks like a nail." Frontier foundation models serve as an easy one-line-of-code hammer, tempting teams to use massive LLMs for basic tasks like string date extraction, vehicle catalog lookups, or text classification. Instead of defaulting to brute-force LLM inference, engineering organizations must assemble a broader toolbox:
- Relying on expensive generalist models to query basic vehicle metadata (make, model, trim, features) wastes compute when internal deterministic VIN-lookup APIs already execute instantly at zero marginal token cost.
- Measurement acts as the fundamental "tape measure" of the entire toolbox; without systematic evaluation, teams cannot determine whether architectural refactors actually improve performance.
---
[11:04](https://www.youtube.com/watch?v=yOQUFKJ0x6w&t=664s) Intentional Agentic Design and Root-Cause Diagnostics
Architecting fast, affordable, and safe agent workflows requires disciplined design principles. First, benchmarking must precede architectural development; optimizing toward a single isolated metric invites metric gaming without real-world utility gains. Second, teams must stop "doing dumb things with LLMs" by offloading structured lookups and simple queries to native software and APIs. Third, architectures must prioritize intentional query routing: placing lightweight intent classifiers in front of request pipelines allows systems to route basic prompts to fast, cheap models (such as Claude Haiku) while reserving frontier models (such as Claude Opus) strictly for high-complexity reasoning.
Luther illustrates the diagnostic gap between broad plausibility and operational utility using an automotive check-engine light example. When fed diagnostic trouble codes, a standard LLM might report a vague "fuel system leak or sensor fault." While technically accurate, this answer is the equivalent of telling a mechanic that the missing object is "somewhere in this house." The true actionable root cause was simply a loose gas cap, which resolves the issue immediately without unnecessary diagnostic turns.
---
[16:10](https://www.youtube.com/watch?v=yOQUFKJ0x6w&t=970s) The Four Traps of AI Benchmarking
Building reliable evaluation benchmarks for complex AI systems reveals four non-obvious traps that derail engineering teams:
```mermaid
flowchart TD
accTitle: Four Traps in Generative AI Benchmarking
subgraph Pitfalls["Benchmarking Pitfalls"]
T1["1. Recognition!= Generation<br/>MCQ tests measure elimination, not generation"]
T2["2. Self-Recognition Artifacts<br/>Models preferentially favor their own synthetic text"]
T3["3. Train/Test Format Mismatch<br/>Rephrased prompts degrade model retrieval"]
T4["4. Contaminated Ground Truth<br/>Human annotation errors & labeling bias"]
end
```
- **Recognition is not equal to generation:** Multiple-choice question (MCQ) benchmarks allow models to eliminate distractors through deductive guessing rather than demonstrating true generative knowledge.
- **Self-recognition artifacts:** Using the same family of models to generate synthetic training sets, run evaluation pipelines, and grade outputs introduces severe bias, as models like Claude systematically favor text generated in their own stylistic distribution.
- **Train and test format mismatch:** Presenting benchmark evaluation prompts in syntactic structures or phrasing that diverges from the training data causes models to fail, despite the underlying domain knowledge being intact.
- **Contaminated ground truth:** Internal data audits revealed that 35% of supposedly "correct" human-labeled reference answers were defective due to human fatigue, domain errors, or labeling conflicts of interest.
---
[19:51](https://www.youtube.com/watch?v=yOQUFKJ0x6w&t=1191s) Calibrating and Pinning the LLM-as-a-Judge
Manual human review does not scale to enterprise evaluation volumes, making automated LLM-as-a-judge pipelines essential. However, deploying an out-of-the-box foundation model as a judge without calibration creates massive variance against expert human ground truth. Deploying an uncalibrated judge is identical to hiring an untrained novice and holding them responsible for production quality assurance without domain training.
```mermaid
flowchart LR
accTitle: LLM as a Judge Calibration and Version Pinning
subgraph Ingestion["1. Calibration Phase"]
SME["Vetted SME<br/>Gold Answers"] --> ALIGN["Align Judge Prompting<br/>& Few-Shot Scoring"]
end
subgraph Execution["2. Production Grading"]
ALIGN --> PIN["Pin Evaluator Version<br/>for Controlled A/B Testing"]
PIN --> RUN["Score System Outputs"]
end
subgraph Evolution["3. Drift & Expansion"]
RUN --> RETRAIN["Iterate Calibration Set<br/>when agent tasks expand"]
RETRAIN -.-> SME
end
```
- Out-of-the-box LLM judges show low alignment when plotted directly against human domain ratings, making it impossible to determine whether the model under test or the judge is failing.
- Effective LLM-as-a-judge implementations calibrate against a curated gold standard provided by vetted Subject Matter Experts (SMEs) until alignment tightens.
- Once calibrated, the judge prompt and model version must be frozen ("pinned") across experiments to maintain an apples-to-apples baseline.
- Whenever agent capabilities or task complexities expand, the calibration process must be re-executed across new test splits to guarantee comprehensive evaluation coverage.
[22:44](https://www.youtube.com/watch?v=yOQUFKJ0x6w&t=1364s) The Model Landscape: Frontier Models, SLMs, and Edge Inference
When mapping model optionality across the landscape, AI architectures generally fall across a three-tier spectrum from massive frontier models to fine-tuned Small Language Models (SLMs) and on-device edge LLMs. On the far left sit the frontier models: while they excel at complex reasoning and nuanced tasks, they incur high per-million-token fees, introduce noticeable latency, and cannot run offline. In the middle tier, SLMs are approximately ten times cheaper and noticeably faster. While they operate best within narrower, domain-specific boundaries, they can run in low-bandwidth or offline environments. On the far right, native mobile and edge LLMs offer virtually zero ongoing inference compute costs and operate entirely offline, making them ideal for predictable, tightly scoped tasks.
```mermaid
flowchart LR
accTitle: AI Model Spectrum from Frontier to Edge
subgraph S1["Frontier LLMs"]
A["Complex reasoning<br/>High cost / High latency<br/>Cloud required"]
end
subgraph S2["Small Language Models (SLMs)"]
B["~10x cheaper & faster<br/>Domain-specific tasks<br/>Works low-bandwidth/offline"]
end
subgraph S3["Edge / Mobile LLMs"]
C["Near-zero compute cost<br/>Low latency<br/>Predictable, offline tasks"]
end
S1 --> S2 --> S3
```
A surprising dynamic emerges when comparing task accuracy across these tiers. While general-purpose frontier LLMs typically score between 70% and 85% accuracy out of the box in specialized enterprise benchmarks, fine-tuned SLMs consistently reach 85% to 95% accuracy because they are trained directly on relevant proprietary domain data. However, fine-tuning requires substantial internal infrastructure, engineering time, and MLOps investment. If an organization lacks dedicated MLOps and infrastructure resources to support continuous model lifecycle experimentation, relying on off-the-shelf frontier models often remains the more pragmatic and cost-effective path.
| Model Tier | Cost Profile | Latency / Bandwidth | Accuracy Benchmark | Best Fit |
|:--- |:--- |:--- |:--- |:--- |
| **Frontier Models** | High ($ / Million tokens) | Slower; requires continuous cloud connection | 70% – 85% out of the box | Complex, open-ended reasoning tasks |
| **Fine-Tuned SLMs** | ~10x cheaper than frontier models | Fast; supports low-bandwidth and offline environments | 85% – 95% on domain tasks | Narrow, domain-specific enterprise workloads |
| **Edge / On-Device** | Essentially free runtime compute | Instant; full offline operation | High on well-scoped tasks | Predictable, repeatable mobile/edge workflows |
---
[25:49](https://www.youtube.com/watch?v=yOQUFKJ0x6w&t=1549s) Model Serving Architectures and Infrastructure Economics
The spectrum of serving infrastructure dictates long-term operating costs and operational complexity, spanning from low-commitment, pay-as-you-go setups on the left to high-fixed-cost dedicated deployments on the right. For teams heavily anchored in cloud platforms like AWS Bedrock, selecting the right deployment pattern depends directly on traffic consistency and volume:
```mermaid
flowchart TD
accTitle: Spectrum of Model Serving Architectures
subgraph LowCommitment["Low Commitment & Variable Cost"]
E["Edge / On-Device<br/>Zero per-call cost, data stays local"]
OD["On-Demand API (e.g., Bedrock)<br/>Pay per token, zero idle costs"]
end
subgraph HighCommitment["High Commitment & Fixed Cost"]
CMI["Custom Model Import<br/>Open-weight models billed by capacity units"]
PE["Provisioned Endpoints<br/>24/7 dedicated instance, full control, fixed bill"]
end
LowCommitment --> HighCommitment
```
- **Edge / On-Device:** Executes compute directly where the user interaction occurs. It carries zero per-call cloud API fees, ensures data never leaves the local environment, and provides optimal reliability for repetitive, lightweight tasks.
- **On-Demand API (e.g., AWS Bedrock Managed):** Charges strictly per token consumed, with zero spend during idle periods. This is the optimal architecture for bursty workloads, irregular traffic spikes, and scheduled batch jobs.
- **Custom Model Import:** Allows teams to bring proprietary fine-tuned weights or open-source foundation models into managed platforms like Bedrock. Infrastructure is billed across capacity unit windows (charged by the minute), providing the most cost-effective hosting option for custom open models.
- **Provisioned Dedicated Endpoints:** Provides a 24/7 dedicated API endpoint with complete configuration control. However, it incurs high, continuous fixed infrastructure costs regardless of whether the cluster is serving queries or sitting completely idle.
---
[28:55](https://www.youtube.com/watch?v=yOQUFKJ0x6w&t=1735s) Proactive Cost Management: Govern, Route, and Account
Cost optimization in enterprise AI is frequently reduced to downstream monitoring, but controlling spend effectively requires shifting from passive accounting to proactive upstream controls. Three distinct verbs define the financial lifecycle: **Govern**, **Route**, and **Account**. Only the first two directly prevent waste; the third merely records past expenditures.
```mermaid
flowchart LR
accTitle: AI Cost Management Framework
subgraph Proactive["Proactive Cost Reductions"]
G["1. GOVERN<br/>Restrict models by team,<br/>use case, and security tier"]
R["2. ROUTE<br/>Automated intelligent query dispatch<br/>(e.g., Haiku vs. Opus)"]
end
subgraph Reactive["Reactive Tracking"]
A["3. ACCOUNT<br/>Descriptive spend reporting &<br/>post-mortem cost auditing"]
end
Proactive --> Reactive
```
Governance establishes structural boundaries so that teams only access the models necessary for their specific problem tier, enforcing security parameters, engineering constraints, and output quality standards. Routing focuses on pairing the right model with the right query. A simple transactional prompt should be dispatched to a lightweight model like Claude Haiku rather than Claude Opus. Because relying on human developers to voluntarily downgrade model selections fails in practice—developers naturally default to the most capable model available—organizations must implement automated, intelligent query routing. Finally, accounting provides descriptive reporting on where budgets were burned after the fact, serving as an audit trail rather than a proactive cost-control mechanism.
---
[31:57](https://www.youtube.com/watch?v=yOQUFKJ0x6w&t=1917s) The Four Gates Before Customizing AI Models
Deciding to fine-tune or train custom models requires passing four sequential evaluation gates. Teams often jump into training prematurely without assessing whether their problem could be resolved using existing techniques or standard foundation models.
```mermaid
flowchart TD
accTitle: The Four Evaluation Gates for Model Customization
G1{"Gate 1: Proprietary Data?<br/>Do you have unique data unavailable to frontier models?"}
G2{"Gate 2: Rigorous Benchmarks?<br/>Can you quantitatively evaluate baseline vs. tuned output?"}
G3{"Gate 3: Simpler Methods Exhausted?<br/>Have prompting, RAG, and smaller models failed?"}
G4{"Gate 4: Sustainable Serving?<br/>Can you afford 24/7 GPU hosting, maintenance, and security?"}
- **Gate 1: Proprietary Data Differentiation:** Determine if your organization possesses specialized knowledge that frontier models have never consumed. For example, Cox Automotive manages vast proprietary automotive service, repair, and transaction records across dealership networks—data detailing specific repairs, vehicle ownership timelines, and maintenance costs that is absent from public web corpora.
- **Gate 2: Comprehensive Benchmarking:** You cannot improve what you cannot measure. Teams must build extensive, rigorous test suites reflecting edge cases across the entire project before training starts.
- **Gate 3: Exhaustion of Classical Optimization:** Custom training should only be pursued after standard techniques—such as advanced prompt engineering, Retrieval-Augmented Generation (RAG), and parameter-efficient small models—have proven inadequate.
- **Gate 4: Long-Term Serving Feasibility:** Custom models forfeit the massive inference, latency, and security optimizations provided by commercial LLM vendors. Organizations must evaluate GPU availability, hosting economics, and ongoing infrastructure maintenance before authoring a single training run.
---
[35:56](https://www.youtube.com/watch?v=yOQUFKJ0x6w&t=2156s) The Hidden Burdens of Custom Model Training
When an engineering organization decides to train and maintain custom model weights, it assumes the entire operational burden that commercial API providers otherwise absorb behind the scenes. Customization introduces four substantial operational liabilities:
```mermaid
flowchart LR
accTitle: Hidden Costs of Custom Training
A["Data Prep is Metered<br/>Every curation loop & turn bills compute"]
B["Data Errors Aren't Patchable<br/>Leaked PII requires full re-training"]
C["The Spotter Walks Off<br/>You own toxicity, bias, & safety guardrails"]
D["Downstream Domino Effects<br/>Fixing one task breaks another capability"]
```
1. **"Data prep is metered":** Every iterative data-cleaning cycle, curation turn, and synthetic generation run generates immediate compute and human engineering costs.
2. **"Bad data isn't patchable":** If personally identifiable information (PII) or corrupted records slip into the training corpus, the weights cannot simply be edited. The team must scrub the source dataset and re-train the model from scratch, wiping out the entire previous financial and compute investment.
3. **"The spotter walks off":** Frontier model providers bake extensive safety, anti-toxicity, and alignment safeguards directly into model weights. Building custom weights means taking full legal and operational responsibility for safety alignment.
4. **Domino effects and regression risk:** Modifying weights to improve performance on one sub-task frequently degrades model capabilities on another, necessitating end-to-end evaluation harnesses at every checkpoint.
---
[38:59](https://www.youtube.com/watch?v=yOQUFKJ0x6w&t=2339s) Synthesis: Balancing Cost, Correctness, and Latency
Navigating enterprise AI design choices comes down to solving the trade-offs between cost, correctness, and latency:
- **Cost Management:** Enforce proactive governance boundaries, implement automated routing systems that reserve frontier models strictly for high-complexity reasoning, and only build custom models when proprietary data justifies the full lifecycle cost.
- **Correctness and Evaluation:** Explicitly define what constitutes a "correct" output—distinguishing between system-level responses and granular component outputs. Calibrate LLM judges and build multi-scenario benchmark suites across every production use case.
- **Latency and Delivery:** Choose the serving path that aligns with user latency expectations, evaluating on-device and edge deployment wherever localized, deterministic execution is possible.
Models will inevitably be rendered obsolete within a year, but the evaluation benchmarks, governance guardrails, and architectural decision frameworks established around them will remain durable assets.
## Notable quotes
> [00:04](https://www.youtube.com/watch?v=yOQUFKJ0x6w&t=4s) beautifully wrong, but still so confident — Udit Luther
> [06:13](https://www.youtube.com/watch?v=yOQUFKJ0x6w&t=373s) if all that you have is a hammer, everything looks like a nail. — Udit Luther
> [11:04](https://www.youtube.com/watch?v=yOQUFKJ0x6w&t=664s) doing dumb things with LLMs — Udit Luther
> [11:04](https://www.youtube.com/watch?v=yOQUFKJ0x6w&t=664s) somewhere in this house. — Udit Luther
> [16:10](https://www.youtube.com/watch?v=yOQUFKJ0x6w&t=970s) Recognition is not equal to generation — Udit Luther
> [35:56](https://www.youtube.com/watch?v=yOQUFKJ0x6w&t=2156s) Data prep is metered — Udit Luther
> [35:56](https://www.youtube.com/watch?v=yOQUFKJ0x6w&t=2156s) Bad data isn't patchable — Udit Luther
> [35:56](https://www.youtube.com/watch?v=yOQUFKJ0x6w&t=2156s) The spotter walks off — Udit Luther
## Mentioned
Ai4, AWS Bedrock, Claude, Claude Haiku, Claude Opus, Cox Automotive, Jodi Blomberg, Porsche, Toyota Camry, Udit Luther
## Video description
Jodi Blomberg - VP AI & ML, Cox Automotive
Udit Luthur - Director, Machine Learning, Cox Automotive
The model landscape has exploded — LLMs, SLMs, fine-tuned models, edge inference, classical ML — and the pressure to ship means most teams default to whatever they are familiar with. But fit-for-purpose AI isn't just about picking the fastest or cheapest option. Observability and explainability are architectural requirements, not afterthoughts, and treating them that way changes every decision downstream.
Fast, cheap, and safe don't have to be tradeoffs- this talk walks through decision frameworks to get all three.
Enterprise AI deployments often falter under runaway expenses, confident hallucinations, and high latency. Drawing on Cox Automotive's engineering practices, Udit Luther argues that long-term viability requires moving past brute-force model usage toward disciplined benchmarking, deterministic tooling, and tiered architectural routing.
Runaway Budgets, Hallucinations, and Latency Bottlenecks (0:04)
Luther identifies three acute pain points confronting enterprise AI leaders: severe bill shock, confident hallucination, and user abandonment driven by high latency. Autonomous agents aggravate cost variance because their open-ended execution loops repeatedly consume context windows and call tools until reaching an exit condition, swinging cloud budgets unpredictably between $40,000 and $4,000,000 without prior notice. Compounding this is a divergence in difficulty: models easily perform complex depreciation calculations yet fail simple common sense, such as an early service agent confidently scheduling customer appointments for nonexistent calendar dates like 'Friday, August 5th.'
Latency functions as a silent killer of user adoption. For Cox Automotive fleet technicians servicing broken-down vehicles on highway shoulders with 70 mph traffic speeding past, repair estimates and cost approvals are time-critical. If an agentic system requires ten-second conversational deliberations, technicians simply abandon the application and place a manual phone call.
Intentional Routing and Deterministic Tooling (6:13)
Defaulting to commercial foundation models for every operational task reflects the 'law of the instrument.' Luther stresses that engineering teams waste compute by using massive LLMs for basic tasks like string extraction or vehicle catalog searches when internal deterministic APIs, such as VIN decoders, execute instantly at zero marginal token cost.
To balance latency and expense, architectures must implement intentional query routing. Placing a lightweight intent classifier upstream allows systems to send routine requests to fast, inexpensive models like Claude Haiku while reserving frontier models like Claude Opus strictly for high-complexity reasoning. Furthermore, systems must prioritize actionable diagnostics over vague plausibility; in check-engine diagnostics, identifying an exact issue like a loose gas cap provides genuine utility, whereas generic summaries of potential sensor faults leave users stranded.
Benchmarking Pitfalls and Calibrated LLM Judges (16:10)
Building robust evaluation harnesses requires steering clear of four common benchmarking traps: multiple-choice formats that measure distractor elimination rather than generative capability; self-recognition bias where models preferentially score text matching their own stylistic distribution; train-and-test format mismatches; and contaminated reference datasets. Internal data audits at Cox Automotive revealed that 35% of supposedly 'correct' human-labeled ground truth answers contained defects caused by human fatigue, domain errors, or labeling bias.
While scaling evaluations requires automated LLM-as-a-judge pipelines, deploying an uncalibrated model yields low alignment with domain experts. Luther emphasizes that an automated judge must be systematically calibrated against curated subject-matter-expert gold standards. Once alignment is achieved, the judge prompt and model version must be frozen—or 'pinned'—across experiments to maintain an uncompromised baseline for A/B testing.
Model Tiering and the Governance of Customization (22:44)
Enterprise architectures operate across a three-tier spectrum: frontier LLMs, fine-tuned Small Language Models (SLMs), and on-device edge models. While frontier models score between 70% and 85% out of the box on specialized benchmarks, fine-tuned SLMs achieve 85% to 95% accuracy on domain tasks at approximately one-tenth the cost. However, fine-tuning requires dedicated MLOps infrastructure. Luther presents four sequential gates before customizing: possessing proprietary data unavailable in public corpora, having rigorous benchmark suites, exhausting prompting and retrieval-augmented generation (RAG), and verifying sustainable long-term GPU hosting economics.
Organizations training custom weights also assume liabilities that commercial API vendors typically absorb. Compute-heavy data preparation cycles are metered, toxicity and safety guardrails must be owned internally, and data errors cannot simply be patched; corrupted training data or leaked personal details necessitate retraining the model from scratch. Consequently, financial sustainability depends on proactive governance and automated query routing rather than retrospective spend accounting.
Luther observes that while individual foundation models will inevitably become obsolete within a year, an organization's proprietary benchmarks, governance guardrails, and architectural routing frameworks remain durable competitive assets.
## Background
本视频由 Ai4 于 2026 年 8 月 14 日发布在 YouTube 平台上。内容基于 Cox Automotive 人工智能与机器学习团队（Jodi Blomberg 与 Udit Luther）的实践经验，围绕模型爆发背景下如何摆脱“默认用最贵模型”的惯性思维，将可观测性与可解释性作为核心架构需求，系统拆解兼顾快速、安全与低成本的 AI 决策框架。
## TL;DR
演讲直面企业落地生成式 AI 时的三大现实痛点——失控的账单、自信的幻觉以及难以容忍的延迟。Cox Automotive 提倡摆脱将前沿巨型模型当作万能“金锤子”的误区，构建涵盖确定性 API、传统代码、轻量小模型（SLM）与端侧推理的复合工具箱。通过基准测试先行、校准 LLM 裁判、实施智能查询分流，并严格执行微调自研的四道准入门槛与防范隐性风险，企业能够构建起在模型迭代中历久弥新的长效工程评估体系。
## Atmosphere & style
整场演讲充满了一线实战工程师的务实与幽默，开场自嘲与犀利洞察迅速拉近了与观众的距离。讲者极善于运用直击痛点的鲜活比喻——从“给叛逆期少年一张无额度信用卡”来形容失控的 Agent，到“高速公路旁等待系统报价的修车工”来强调低延迟的生死攸关，将晦涩的架构权衡讲解得生动透彻。整体叙事张弛有度，既有对行业盲目追新现象的冷静反思，又洋溢着经过真实业务淬炼后的工程智慧与落地底气。
## Key takeaways
- [00:04](https://www.youtube.com/watch?v=yOQUFKJ0x6w&t=4s) 企业在生成式 AI 落地过程中集中面临三大现实挑战：难以预测的高昂账单（成本）、文笔优雅却自信笃定的事实错误（幻觉与安全）以及超出用户耐受度的响应延迟（速度）。
- [01:50](https://www.youtube.com/watch?v=yOQUFKJ0x6w&t=110s) 自主智能体（Agent）具备“不惜代价达成目标”的特性，容易陷入死循环并盲目调用工具，加上配套基建、可观测性与持续评测开销，会导致账单方差失控飙升。
- [03:47](https://www.youtube.com/watch?v=yOQUFKJ0x6w&t=227s) 大模型与人类的认知难度存在错位：模型能轻松推导高深数学公式，却极易在常识推理上翻车；而在开放式生成任务中，缺乏唯一定义的真实基准标签（Ground Truth）给评测带来了巨大挑战。
- [06:13](https://www.youtube.com/watch?v=yOQUFKJ0x6w&t=373s) 在紧急或高频的一线业务场景中，延迟往往是决定产品生死的关键指标，低延迟必须作为前置核心架构需求进行设计，而非后期修补。
- [07:57](https://www.youtube.com/watch?v=yOQUFKJ0x6w&t=477s) 开发者应摒弃“凡事皆用前沿 LLM”的“金锤子”思维，构建包含轻量级小模型（SLMs）、确定性 API 和传统代码在内的多元化工具箱，以零边际成本解决简单任务。
- [11:04](https://www.youtube.com/watch?v=yOQUFKJ0x6w&t=664s) Agent 设计必须遵循三项核心原则：基准测试先行且严防指标单一化、坚决拒绝用 LLM 代替已有的确定性 API、在入口处利用轻量分类器按复杂度分级路由模型。
- [13:51](https://www.youtube.com/watch?v=yOQUFKJ0x6w&t=831s) 基准评测暗藏四大陷阱（识别不等于生成、自认伪影偏差、格式错位及标注污染），审计显示人工标注的“标准答案”甚至可能包含高达 35% 的缺陷与噪声。
- [19:51](https://www.youtube.com/watch?v=yOQUFKJ0x6w&t=1191s) 开箱即用的大模型裁判（LLM as a Judge）评分高度发散，必须通过领域专家（SME）精标数据集进行对齐校准，并在特定版本上锚定（Pin）以确保长期评估的一致性。
- [22:44](https://www.youtube.com/watch?v=yOQUFKJ0x6w&t=1364s) 针对特定领域微调的小模型（SLM）在准确率上可达 85%–95%，显著超越通用前沿大模型，且推理成本大幅降低，但团队需将搭建 MLOps 体系的综合成本纳入 TCO 考量。
- [25:49](https://www.youtube.com/watch?v=yOQUFKJ0x6w&t=1549s) 模型部署应根据业务吞吐量与承诺成本进行权衡：从零调用费的端侧部署、适合突发流量的按需托管 API，到按容量计费的自定义模型导入和 24/7 预置专属终端节点。
- [28:55](https://www.youtube.com/watch?v=yOQUFKJ0x6w&t=1735s) 成本管控依赖治理（访问权限控制）、路由（自动化智能复杂度分发）与核算（事后被动审计）三大支柱，必须坚持主动防御，杜绝开发者无差别依赖旗舰模型的习惯。
- [31:57](https://www.youtube.com/watch?v=yOQUFKJ0x6w&t=1917s) 启动自研或微调前必须通过四道准入门槛：是否拥有独家高价值私有数据、是否建立全场景基准评测、是否穷举排除了 Prompting/RAG 等轻量手段、以及是否算清了长期 Serving 的运维经济账。
- [35:56](https://www.youtube.com/watch?v=yOQUFKJ0x6w&t=2156s) 自建模型需全盘承担四大隐性风险：持续的数据准备费用、脏数据无法局部修补（泄漏 PII 只能重训）、失去厂商内置的安全护栏，以及局部改动引发全局性能衰退的多米诺效应。
- [39:03](https://www.youtube.com/watch?v=yOQUFKJ0x6w&t=2343s) 具体模型注定会随着技术迭代而过时，但围绕业务清晰定义“何为优质表现”并建立起来的基准评测、质量门槛与决策框架将历久弥新。
## Detailed walkthrough
### [00:04](https://www.youtube.com/watch?v=yOQUFKJ0x6w&t=4s) 开场与生成式 AI 落地面临的三大现实痛点
演讲由 Cox Automotive 负责生成式 AI 评估与模型管理的机器学习总监 Udit Luther 开场。他幽默地向观众致歉，解释原本该同台的团队主管 Jodi Blomberg 因不可抗力未能到场，"大家只能勉为其难跟我待在一起了"。Udit 开宗明义地指出，企业在过去几个月中全面推行生成式 AI 时，往往都会遭遇三个直击灵魂的现实挑战，并邀请在场听众产生共鸣。
第一个痛点是令人瞠目结舌的账单——许多团队在月底查看 AI 消费时，眼睛都快瞪出来了；第二个痛点是模型展现出的“漂亮而自信的错误”，AI 能够以极其笃定、权威且文笔优雅的口吻给出完全错误的答案，以至于让人根本察觉不到它在胡说八道；第三个痛点则是日益凸显的延迟焦虑，随着用户对 AI 越来越熟悉，曾经让人惊叹的技术如今哪怕多让用户等待 10 秒钟，都会引发极大的不耐烦。这三者共同构成了企业部署 AI 过程中最棘手的成本、安全与速度难题。
- **核心议题**：Cox Automotive 内部如何系统性思考并消除成本激增、置信幻觉与高延迟这三大落地瓶颈。
---
### [01:50](https://www.youtube.com/watch?v=yOQUFKJ0x6w&t=110s) 成本失控：像给十几岁青少年一张没有额度的信用卡
在深入探讨成本时，Udit Luther 指出自主智能体（Agent）的运作逻辑往往与成本控制背道而驰。Agent 的天性是“不惜一切代价完成既定目标”，它们本身没有金钱与开销的概念，甚至即便有也毫不关心。为了达成目的，Agent 会在任务中反复兜圈子，一遍又一遍地加载并读取相同的上下文，甚至盲目调用与当前任务毫不相干的外部工具。这种不断自我循环的调用使得推理费用成倍翻番。
更为严峻的是，运行 Agent 的真实成本绝不仅限于模型推理（Inference）。在生产环境中托管 Agent，企业还必须承担庞大的配套基建费用、全链路可观测性（Observability）支出、端到端评估（Evaluations）体系的开销以及安全与合规审计成本。Udit 打了一个生动的比方：给 Agent 赋予调用权限，就像“给十几岁的叛逆期青少年一张信用卡”，这种不可控性足以让任何一位首席财务官（CFO）抓狂。更让财务部门无法制定预算的是其惊人的方差——单月账单可能在 4 万美元到 400 万美元之间剧烈波动。Udit 调侃道，唯一值得欣慰的或许是，Agent 最终大概率真能把任务办完，而叛逆期的青少年可不一定。
- **Agent 成本倍增机制**：死循环上下文重读、无效工具调用叠加。
- **全生命周期隐藏开销**：基础设施托管、监控可观测性、安全防护体系与持续评估成本。
- **财务方差失控**：从几万美元瞬间飙升至数百万美元，彻底击穿传统 IT 预算模型。
---
### [03:47](https://www.youtube.com/watch?v=yOQUFKJ0x6w&t=227s) 似是而非的准确度与基准测试的天然困境
探讨第二个问题时，Udit 展示了一张“人类难度 vs 大语言模型难度”的对比坐标图。现实中二者之间几乎毫无规律可循：对人类来说极度复杂的专业任务，LLM 可能轻而易举就能搞定；反之，人类一眼便知的生活常识，模型却会惨遭绊倒。例如在 Cox Automotive 的业务场景中，模型能够洋洋洒洒撰写出极高水准的行业分析长文，甚至完美推导汽车复杂的折旧曲线数学公式；但在同一个任务中，它却能煞有介事地把一台 V8 发动机强行塞进一辆 4 缸丰田凯美瑞（Toyota Camry）的配置单里。
类似令人啼笑皆非的案例屡见不鲜。团队曾开发过帮助用户预约看车购车的智能客服 Agent，当用户提出预约时，模型以无比肯定的语气回复：“已帮您预约在本周五，即 8 月 5 日”。然而在当时的真实日历上，8 月 5 日其实是次日周六。这种在细微事实与逻辑上的“笃定翻车”，给企业带来了极大的声誉风险。更棘手的是，在开放式生成场景下，传统机器学习所依赖的“真实基准标签”（Ground Truth）极难定义。当用户提问“请推荐几款适合家庭使用的汽车”，如果模型给出了一辆保时捷（Porsche），这到底算不算错？这种主观模糊性决定了建立客观基准（Benchmarking）和评估机制绝非易事，但它却是系统安全上线的唯一命门。
```mermaid
flowchart LR
accTitle: 模型与人类能力认知反差
subgraph S1["人类视角的认知难度"]
H1["复杂折旧曲线数学公式<br/>(人类觉得极难)"]
H2["常识：凯美瑞是4缸车<br/>(人类觉得极易)"]
end
subgraph S2["LLM 实际处理表现"]
M1["轻松给出严密数学推导<br/>(模型完成度极高)"]
M2["自信把 V8 发动机配给凯美瑞<br/>(模型常识性翻车)"]
end
H1 --> M1
H2 --> M2
```
- **认知错位**：LLM 擅长复杂的长文本与数理抽象，却极易在常识推理、日历对齐和具体规格实体上犯低级错误。
- **开放式文本的 Ground Truth 危机**：主观性、发散性回答使得简单的精确匹配规则完全失效。
---
### [06:13](https://www.youtube.com/watch?v=yOQUFKJ0x6w&t=373s) 延迟即死局：高速公路旁等待报价的卡车维修工
延迟（Latency）往往是架构师在方案设计最后阶段才会考虑的指标，但在终端用户眼中，它却是最先招致抱怨、甚至直接决定产品生死的要素。Udit 举了一个生动的重型车队服务一线场景：Cox 旗下拥有大型商用卡车救援服务，一名浑身油污的维修技师正站在洲际公路的路肩上抢修卡车，身旁就是以 70 英里时速呼啸而过的车流。
这名技师调出内部工具，正在焦急等待系统给出故障维修预估并获取审批，以便尽快抡起扳手开始修车。对身处险境的工人而言，他根本不需要模型吐出一篇文采斐然、构思精妙的万字长文，他只想要一个准确、极速的判断，好让他修完车立刻撤离危险的高速路边缘。如果智能系统在每一轮对话中都要耗费 10 秒钟去思考和生成，这位技师会毫不犹豫地关掉软件，直接给调度中心拨打电话。Udit 强调，低延迟必须是一开始就纳入考量的核心架构需求，而不是系统上线后再去缝补的补丁。
---
### [07:57](https://www.youtube.com/watch?v=yOQUFKJ0x6w&t=477s) 摆脱“金锤子”思维：构建更丰富的 AI 工具箱
为什么当前企业在应用 AI 时会集中爆发成本、幻觉与延迟这三大矛盾？Udit 援引俗语解释道：“如果你手里只有一把锤子，你看什么都像钉子。”在当前的技术热潮中，那把无所不能的“金锤子”就是前沿大语言模型（Frontier LLMs）。开发者只需写一行调用代码，模型就能开箱即用，面对任何输入都能硬着头皮给出解答。
但这并不意味着凡事都该交给 LLM。Udit 连续发问反思技术滥用现象：我们真的需要动用一个巨型 LLM 去做基础文本分类吗？真的需要让 LLM 去字符串里提取日期吗？甚至，真的需要专门调用 LLM 去查询天气吗？看一眼手机天气或调个 API 就能秒级免费解决的事情，很多人却在用最昂贵的大模型去烧 Token。企业真正需要的是一个更大的“百宝箱”——里面不仅有前沿模型，还应当囊括轻量级小模型（SLMs）、确定性 API，以及由资深工程师编写的、经受住数十年检验的传统代码。这类代码一旦写定，不仅运行飞快，而且边际调用成本为零。
```mermaid
flowchart TD
accTitle: 混合式 AI 架构任务分发流程
UserQuery["用户原始请求"] --> Classifier{"轻量级路由器 / 分类器"}
Classifier -- "简单实体查询 / 日期提取" --> Deterministic["内部 API 与传统代码<br/>(零推理费 / 毫秒级)"]
Classifier -- "常规通识与低复杂度对话" --> SLM["轻量级小模型 / Haiku 等<br/>(低成本 / 极快响应)"]
Classifier -- "复杂多步推理与综合创作" --> Frontier["前沿巨型模型 / Opus 等<br/>(处理高难度任务)"]
```
- **“金锤子”典型滥用场景**：简单单标签分类、正则级日期解析、实时静态天气查询。
- **多元化工具箱的组成**：专业小模型（SLM）、企业内部确定性 API、传统算法与规则代码。
---
### [11:04](https://www.youtube.com/watch?v=yOQUFKJ0x6w&t=664s) Agent 设计三原则：测量先行、拒绝盲目调用与分级路由
基于构建百宝箱的思考，Udit 提出了 Agent 架构设计的核心三大原则。但在此之前，他强调必须有一把“卷尺”贯穿始终——即严密的度量（Measurement）体系。没有量化标尺，工具再多也毫无用处，因为你根本无法证明所做的架构优化究竟是提升了系统还是劣化了体验。
在 Agent 设计原则方面，Udit 总结了三条铁律：
1. **基准测试先行（Benchmarking comes first）**：严禁把测试集混入训练数据，这在数据科学中是欺骗行为。同时，绝不能单一盲目追求某一个孤立指标（如单纯的满意度得分）。一旦指标单一化，工程师和模型自身都会想方设法去“刷高”该指标，导致整体系统目标发生严重形变。
2. **拒绝用 LLM 干蠢事（Don't do dumb things with LLMs）**：在 Cox Automotive 内部，公司沉淀了极具价值的车型目录与车辆识别码（VIN）解码 API，输入一串 VIN 码就能毫秒级返回精确的品牌、型号、年款、配置套件等所有细节。然而不少内部系统早期依然直接让 LLM 盲猜车型配置，不仅白白烧钱，还增加了幻觉风险。
3. **面向低成本与高速度进行针对性设计（Design for low cost and speed）**：在架构前置设计中保持审慎与克制。例如可以在系统入口处布置一个轻量级分类器，识别用户意图的复杂度。如果是常规查询，坚决路由给 Claude Haiku 级别的小模型，只有遇到极其晦涩复杂的多步推理时，才放行给 Claude Opus 这样的旗舰模型，从而在保障性能的同时大幅压降成本与响应时间。
---
### [13:51](https://www.youtube.com/watch?v=yOQUFKJ0x6w&t=831s) 评测基准的实战深潜：从油箱盖松动到四大评测陷阱
在实操层面，构建基准测试极其困难。Udit 举了一个汽车维修诊断的现实案例：一辆车进厂报修，发动机故障灯亮起，车辆自诊断代码被输入给 LLM。LLM 回复称“该车存在燃油系统泄漏或传感器故障”。从技术上讲，模型的回答并没错，确实属于相关子系统，但这就像是在大厦失火时告诉消防员“东西就在这栋楼或这个房间的某个地方一样”，毫无实质帮助。而经验丰富的技师排查出的根本原因其实极其微小：只是汽车的油箱盖（Gas Cap）没有拧紧。模型模棱两可的正确，会导致维修技师在公路边多浪费几个小时排查无辜零件，而具体的答案能让人秒级排除故障。
在自主研发与模型微调的探索中，Cox 团队总结出了四大典型评测暗坑：
| 评测陷阱类型 | 核心表象与风险机制 | 应对与反思策略 |
|:--- |:--- |:--- |
| **识别不等于生成**<br>*(Recognition!= Generation)* | 使用多项选择题（MCQ）评测模型，模型可通过“排除法”选出正确选项，而实际上根本无法在无提示下自主生成答案。 | 针对开放式 Agent 任务，严禁用选择题准确率代替真实生成能力评测。 |
| **自认伪影偏差**<br>*(Self-recognition Artifacts)* | 使用同一模型家族（如 Claude）既生成训练数据又充当评估裁判，模型对自身句式、风格有天然的偏好与高分倾向。 | 避免使用同源模型自产自评，防止未意识到的系统性评分偏差。 |
| **训练与测试格式错位**<br>*(Train/Test Format Mismatch)* | 训练数据采用某种特定句式或结构，测试时换成另一种表述方式，模型便会意外溃败，无法泛化其已有知识。 | 丰富多样化测试用例的语义形态，验证知识泛化而非句式过拟合。 |
| **被污染的真实标注**<br>*(Contaminated Ground Truth)* | 人工标注的数据并不绝对正确；Cox 在对人类“标准答案”进行深度审计时，震惊地发现高达 35% 的标注本身存在缺陷。 | 建立多轮审核过滤机制，并防范人类标注员因利益冲突（不希望被自动化系统取代）故意引入的噪声。 |
- **关键审计发现**：所谓由人类专家标注的“基准真值”（Ground Truth）中，竟然包含高达 **35%** 的瑕疵与错误。
---
### [19:51](https://www.youtube.com/watch?v=yOQUFKJ0x6w&t=1191s) LLM as a Judge 的校准法则：告别“开箱即用”与主观偏差
面对海量生成文本，完全依赖人工实时审核在大规模业务中根本不具备可行性，使用“大模型作为裁判”（LLM as a Judge）成为了行业标配路径。然而，Udit 展示的散点图揭示了一个残酷现实：若直接使用未经校准、开箱即用的商用大模型充当裁判，其打分与人类资深专家的评估结果散落成一片毫无规律的弥散星云，根本分不清究竟是模型判错了还是人类看走了眼。这就像直接把一个刚招进公司、毫无业务培训与实战经验的新人推到质检总监的位置上，失败是注定的。
为此，Cox 探索出了科学的 LLM 裁判校准流程。首先，绝不能让裁判模型凭空评判，而必须使用经过严格背对背交叉审查的领域专家（SME）精标数据集对其进行强制对齐校准。在不断微调与 Prompt 约束下，让裁判模型的打分逐步收敛到高度吻合专家标准的直线分布上。一旦校准完成，团队会在这一特定版本上“锚定”（Pin）裁判模型，确保后续不同架构、不同开源/闭源模型在做对比实验时拥有绝对统一的标尺。Udit 最后提醒，这种校准不是一劳永逸的；随着业务 Agent 承接任务边界的拓展，裁判模型必须定期重新校准，以维持在新型复合场景下的评测信度。
```mermaid
flowchart TD
accTitle: LLM 裁判校准与锚定闭环
A["开箱即用 LLM 裁判<br/>(评分分散/缺乏业务常识)"] --> B["引入 SME 领域专家精标数据集"]
B --> C["针对性对齐与校准训练"]
C --> D["对齐验证：与专家判定分布吻合"]
D --> E["版本锚定 (Pinned Judge)<br/>用于全产品线横向公平对照"]
E --> F{"Agent 拓展新业务场景?"}
F -- "是" --> B
F -- "否" --> G["持续稳定评估"]
```
### [22:44](https://www.youtube.com/watch?v=yOQUFKJ0x6w&t=1364s) 模型选型光谱：前沿大模型、小语言模型与端侧模型
在探讨智能体（Agentic）设计原则之后，模型本身的选择机制（Model Optionality）同样关键。从技术与架构的光谱来看，模型选型可以清晰地划分为三个层级：
```mermaid
flowchart LR
accTitle: 模型选型光谱对比
subgraph S1["前沿大模型 (Frontier Models)"]
A["高推理能力 / 复杂任务"]
A1["按百万 Token 计费 / 依赖网络"]
end
subgraph S2["小语言模型 (SLMs)"]
B["领域专精 / 细分任务"]
B1["成本低 10 倍 / 弱网或离线"]
end
subgraph S3["端侧 / 原生移动模型 (Edge Models)"]
C["确定性 / 结构化任务"]
C1["单次调用近乎免费 / 极低延迟"]
end
S1 --> S2 --> S3
```
最左侧是**前沿大模型（Frontier Models）**，其调用成本按每百万 Token 计费，响应速度相对较慢，且必须依赖互联网连接，但它们在处理高度复杂、需要深度逻辑推理的任务时不可或缺。中间层是**小语言模型（SLMs）**，其调用成本比前沿模型便宜约 10 倍，速度更快，并支持在低带宽甚至离线场景下运行；SLM 更加偏向特定领域，适用于更窄、更明确的任务。最右侧则是**端侧或原生移动端模型（Edge / Native Mobile LLMs）**，由于直接运行在终端设备上，其调用成本基本为零，具备极快的响应速度和无网运行能力，最适合执行可预测、定义明确的任务。
在准确率方面，实际测试展现出了反直觉的结论：通用前沿大模型在开箱即用时的基准准确率约为 70% 至 85%；然而，一旦针对特定领域对 SLM 进行微调，由于模型注入了精准的相关业务数据并围绕其进行推理，其准确率往往能攀升至 85% 到 95%，显著超越通用的前沿大模型。
不过，团队在选择微调路线时必须保持清醒。微调的高准确率背后依托着成熟的基础设施、工程资源和 MLOps 团队支持。对于缺乏相应研发资源与基础设施投入的团队，搭建 MLOps 体系本身的隐形成本极其高昂，此时直接采用前沿模型开箱即用的能力反而是更经济、更省力的务实选择。
- **前沿模型**：高推理复杂度，按 Token 付费，强网络依赖，基准测试准确率约 70%–85%。
- **微调 SLM**：成本降低约 10 倍，支持低带宽/离线，注入领域数据后准确率可达 85%–95%。
- **端侧 LLM**：边际调用成本为零，延迟极低，适用于边界明确的高频固定任务。
- **决策前提**：计算微调收益时，必须将工程人力与 MLOps 平台建设成本纳入总拥有成本（TCO）。
---
### [25:49](https://www.youtube.com/watch?v=yOQUFKJ0x6w&t=1549s) 模型部署与推理架构的权衡考量
确定模型类型后，核心问题转向模型的部署与服务（Serving）架构。部署方案的光谱由业务吞吐量、固定成本承诺以及付费模式共同决定：左端代表低交易量、零保底承诺与按量付费；右端则代表高业务量、高固定成本以及闲置期持续计费。
```mermaid
flowchart TD
accTitle: 模型部署与推理架构决策路径
subgraph Spectrum["部署模式光谱（从低承诺到高专属）"]
E["端侧 / 设备本地 (Edge)"]
D["按需托管 API (On-Demand API)"]
C["自定义模型导入 (Custom Model Import)"]
P["预置专属终端节点 (Provisioned Endpoint)"]
end
E -->|零调用费 / 数据不离端| D
D -->|按 Token 计费 / 适合突发流量| C
C -->|按容量时间计费 / 适合开源权重| P
P -->|24/7 专属资源 / 适合持续超高吞吐| Spectrum
```
1. **端侧部署（Edge / On-Device）**：模型直接运行在工作负载发生的本地设备。优势在于没有单次调用费用，且原始数据完全保留在本地安全域内，最适合轻量、高频且重复度高的任务。
2. **按需 API（On-Demand API）**：以 AWS Bedrock 托管服务为例，完全按 Token 用量计费，系统在闲置期间不产生任何计算费用。这种模式最适合批处理任务或流量呈现潮汐式、突发性特征的业务场景。
3. **自定义模型导入（Custom Model Import）**：允许企业将自研训练模型或开源权重（Open-weight models）直接导入托管平台（如 AWS Bedrock）。平台通过“容量单元窗口”（Capacity Unit Windows）按分钟收取底层预留算力的费用，这是部署定制化开源模型成本最低的路径之一。
4. **预置专属终端节点（Provisioned Endpoint）**：企业独占专属 API 实例，享有 24/7 全天候计算资源保障。虽然拥有最高的控制权与定制自由度，但无论业务流量是否发生，企业都必须承担持续运转的基础设施固定费用。
---
### [28:55](https://www.youtube.com/watch?v=yOQUFKJ0x6w&t=1735s) 成本控制的三大支柱：治理（Govern）、路由（Route）与核算（Account）
在构建 AI 与智能体系统时，企业经常混淆成本管控的概念。有效的成本控制依赖于三大支柱，其中前两项属于主动干预，而最后一项仅属于被动分析：
| 成本管控支柱 | 介入性质 | 核心机制与实施策略 |
|:--- |:--- |:--- |
| **治理 (Govern)** | 主动防范 | 建立访问控制策略，按业务场景、安全规范与工程要求限制模型权限，严禁全员无差别访问顶配模型。 |
| **路由 (Route)** | 主动优化 | 实施“最合适模型匹配最合适任务”，通过自动化智能路由器根据查询复杂度动态分发请求，避免人工低效选择。 |
| **核算 (Account)** | 被动分析 | 事后复盘资金消耗账单与不必要开支，提供描述性审计支持，无法从源头阻止超支。 |
在治理层面，Cox Automotive 严格限制各业务团队对顶尖模型的调用权限，只有真正处于探索前沿创新且对推理能力有刚需的项目，才会获批接入最高阶的模型。
在路由机制上，实践表明仅靠对开发人员进行“成本教育”是远远不够的。如果把模型选择权完全交给个人，大多数人会产生依赖心理，“倾向于把所有任务无论难易统统扔给 Claude Opus 这样最强大的模型”。因此，必须在架构层引入**自动化智能查询路由机制（Automated Intelligent Routing）**，由系统在后台根据输入特征自动评估复杂度，例如将简单指令分发给轻量的 Claude Haiku，将深度推理任务分配给 Opus，彻底剥离人工选择的负担。
至于最后的核算环节，它仅能作为事后复盘与审计工具，展示资金流向与低效开支。“核算只能告诉你过去发生了什么，它本身无法在运行中控制成本”，必须坚持治理与路由先行的主动防御策略。
---
### [31:57](https://www.youtube.com/watch?v=yOQUFKJ0x6w&t=1917s) 自定义与微调模型的四道准入门槛
在决定踏入微调或自建模型之前，团队必须依次通过四道严格的决策门槛（Gates），以避免陷入盲目自研的泥潭：
```mermaid
flowchart TD
accTitle: 模型自定义四道准入门槛
G1{"门槛 1: 拥有独家私有数据吗？"}
G2{"门槛 2: 建立全场景基准评测了吗？"}
G3{"门槛 3: 传统基线方法彻底失效了吗？"}
G4{"门槛 4: 算得清长期 Serving 综合成本吗？"}
Pass["正式启动模型自研 / 微调"]
Stop["退回：采用开箱即用模型 / 提示词工程 / RAG"]
G1 -->|否| Stop
G1 -->|是| G2
G2 -->|否| Stop
G2 -->|是| G3
G3 -->|否| Stop
G3 -->|是| G4
G4 -->|否| Stop
G4 -->|是| Pass
```
- **第一道门：独家高价值数据资产校验**
核心在于明确现有开箱即用模型是否确实无法满足需求，以及企业自身是否拥有前沿大模型尚未涉足的独特数据。以 Cox Automotive 为例，由于拥有广泛服务汽车经销商的专有软件体系，公司积累了海量的真实修车与维保交易记录，包括具体车型、故障发生时间、维修项目及实际开销。这类涵盖汽车全生命周期拥有成本（TCO）的数据完全不对公网开放，正是利用微调为模型注入专属领域认知的高价值源泉。
- **第二道门：全面完备的基准评测体系（Benchmarking）**
必须沉下心做扎实的工程苦工，为目标场景搭建覆盖全面的基准数据集。如果没有基准评测，团队便无法量化任何微调带来的实际提升，所有的“优化”都会沦为主观臆断。
- **第三道门：基础优化手段的穷举验证**
在启动微调前，必须确认是否已经充分尝试并排除了前几年成熟的轻量级优化技术，包括提示词工程（Prompting）、检索增强生成（RAG）以及选用现成的高性能轻量模型。只有证明这些低成本手段均无法达到指标时，才有理由推进至模型定制。
- **第四道门：推理服务（Serving）的长期经济账**
部署服务绝非简单的采购几张 GPU，它涉及底层基础设施运维、安全边界管理以及性能优化。前沿模型厂商在服务层所做的海量底层推理优化，自建团队都必须全盘自行承接。这一考量必须在训练开始前彻底理清，而不是在模型练成之后才发现无力支撑上线。
---
### [35:56](https://www.youtube.com/watch?v=yOQUFKJ0x6w&t=2156s) 自研微调的隐形成本与风险
选择自建与微调模型，意味着企业必须全盘接管由自身掌控模型带来的复杂连带责任。这些风险可以总结为四个典型的隐性技术陷阱：
```mermaid
flowchart TD
accTitle: 自建与微调模型的四大隐形成本与风险
subgraph HiddenRisks["自建模型的四大隐形技术陷阱"]
R1["数据准备持续计费<br/>(Prep is Metered)<br/>每一次清洗迭代都是真金白银"]
R2["脏数据不可修补<br/>(Bad Data Isn't Patchable)<br/>泄漏 PII 意味着整体推倒重训"]
R3["安全护栏全盘自理<br/>(The Spotter Walks Off)<br/>失去厂商内置权重级对齐与安全保障"]
R4["训练改动多米诺效应<br/>(Domino Effects)<br/>修复局部问题极易诱发全局指标衰退"]
end
```
1. **数据准备持续计费（Prep is metered）**：清洗、标注和构建微调数据的每一次循环迭代都需要消耗昂贵的算力与工程人力，每次微调改动都会产生新的账单。
2. **脏数据无法打补丁（Bad data isn't patchable）**：在 Cox Automotive 的实际演练中，若训练集意外渗漏了个人身份信息（PII），没有任何手段能在模型权重中进行局部“修补”；唯一的解法是将污染数据剔除后从头完整重新训练，这意味着先前的所有训练算力投入全部归零。
3. **安全护栏全盘自理（The spotter walks off）**：主流前沿大模型厂商在权重训练阶段便内置了大量的安全对齐（Safety Alignment）和防毒性（Toxicity）机制。选择自建微调模型，就意味着失去了原厂的安全监护，模型合规与输出安全的所有技术责任必须由企业完全自行承担。
4. **多米诺骨牌效应（Domino effects）**：在微调过程中针对某一特定子任务或特定输出进行的针对性调整，往往会产生不可预知的连带副作用，导致模型在其他任务上的表现崩塌。因此，完备的回归评估系统（Evaluation Pipeline）必须贯穿训练始终，以便实时监控模型整体性能的健康度。
从输入与输出的结构化程度来看：输入与输出越非结构化，企业需要自行承载的研发、维护与容错成本就越呈现指数级上升。
---
### [39:03](https://www.youtube.com/watch?v=yOQUFKJ0x6w&t=2343s) 总结与落地行动指南
构建高效、经济且安全的 AI 体系，本质上是在**成本（Cost）**、**正确性（Correctness）**与**延迟（Latency）**三者之间建立理性的工程平衡：
- **成本把控**：严格落实模型治理，构建智能路由机制，将非复杂任务导流至小模型，仅将前沿大模型保留给真正高难度的推理挑战；严谨评估自建模型的全生命周期经济模型。
- **正确性定义**：明确业务场景中“正确”的真正含义——究竟是评估整个系统级流水线的最终表现，还是特定局部组件的推理准确度？建立全场景的基准测试集并校准大模型评委（Judges）。
- **延迟与部署优化**：依据时延诉求匹配部署模式，对于确定性强、高频重复的任务，端侧部署是极具潜力的可行解。
对于任何企业而言，今天就可以立即启动的行动是：审视团队目前已上线的 AI 功能，清晰书写并定义该场景下“何为优质表现（What does good mean）”，并据此构建可量化的评测流程。正如分享所言：“**今天你所使用的具体模型，一年之后注定会过时淘汰；但你围绕业务建立起来的评测基准、质量门槛与决策框架，却能历久弥新。**”
## Notable quotes
> [00:04](https://www.youtube.com/watch?v=yOQUFKJ0x6w&t=4s) 大家只能勉为其难跟我待在一起了 — Udit Luther
> [01:50](https://www.youtube.com/watch?v=yOQUFKJ0x6w&t=110s) 给十几岁的叛逆期青少年一张信用卡 — Udit Luther
> [07:57](https://www.youtube.com/watch?v=yOQUFKJ0x6w&t=477s) 如果你手里只有一把锤子，你看什么都像钉子。 — Udit Luther
> [11:04](https://www.youtube.com/watch?v=yOQUFKJ0x6w&t=664s) Benchmarking comes first — Udit Luther
> [11:04](https://www.youtube.com/watch?v=yOQUFKJ0x6w&t=664s) Don't do dumb things with LLMs — Udit Luther
> [11:04](https://www.youtube.com/watch?v=yOQUFKJ0x6w&t=664s) Design for low cost and speed — Udit Luther
> [13:51](https://www.youtube.com/watch?v=yOQUFKJ0x6w&t=831s) Recognition != Generation — Udit Luther
> [28:55](https://www.youtube.com/watch?v=yOQUFKJ0x6w&t=1735s) 倾向于把所有任务无论难易统统扔给 Claude Opus 这样最强大的模型 — Udit Luther
> [28:55](https://www.youtube.com/watch?v=yOQUFKJ0x6w&t=1735s) 核算只能告诉你过去发生了什么，它本身无法在运行中控制成本 — Udit Luther
> [35:56](https://www.youtube.com/watch?v=yOQUFKJ0x6w&t=2156s) Prep is metered — Udit Luther
> [35:56](https://www.youtube.com/watch?v=yOQUFKJ0x6w&t=2156s) Bad data isn't patchable — Udit Luther
> [35:56](https://www.youtube.com/watch?v=yOQUFKJ0x6w&t=2156s) The spotter walks off — Udit Luther
> [35:56](https://www.youtube.com/watch?v=yOQUFKJ0x6w&t=2156s) Domino effects — Udit Luther
> [39:03](https://www.youtube.com/watch?v=yOQUFKJ0x6w&t=2343s) 今天你所使用的具体模型，一年之后注定会过时淘汰；但你围绕业务建立起来的评测基准、质量门槛与决策框架，却能历久弥新。 — Udit Luther
## Mentioned
Cox Automotive, Udit Luther, Jodi Blomberg, Ai4, Claude Haiku, Claude Opus, AWS Bedrock, Toyota Camry, Porsche
## Video description
Jodi Blomberg - VP AI & ML, Cox Automotive
Udit Luthur - Director, Machine Learning, Cox Automotive
The model landscape has exploded — LLMs, SLMs, fine-tuned models, edge inference, classical ML — and the pressure to ship means most teams default to whatever they are familiar with. But fit-for-purpose AI isn't just about picking the fastest or cheapest option. Observability and explainability are architectural requirements, not afterthoughts, and treating them that way changes every decision downstream.
Fast, cheap, and safe don't have to be tradeoffs- this talk walks through decision frameworks to get all three.
[Source](https://www.youtube.com/watch?v=yOQUFKJ0x6w)
Source quotes
[0:04] Udit Luther: 大家只能勉为其难跟我待在一起了
[1:50] Udit Luther: 给十几岁的叛逆期青少年一张信用卡
[7:57] Udit Luther: 如果你手里只有一把锤子，你看什么都像钉子。
[11:04] Udit Luther: Benchmarking comes first
[11:04] Udit Luther: Don't do dumb things with LLMs
[11:04] Udit Luther: Design for low cost and speed
[13:51] Udit Luther: Recognition != Generation
[28:55] Udit Luther: 倾向于把所有任务无论难易统统扔给 Claude Opus 这样最强大的模型
[28:55] Udit Luther: 核算只能告诉你过去发生了什么，它本身无法在运行中控制成本
[35:56] Udit Luther: Prep is metered
[35:56] Udit Luther: Bad data isn't patchable
[35:56] Udit Luther: The spotter walks off
[35:56] Udit Luther: Domino effects
[39:03] Udit Luther: 今天你所使用的具体模型，一年之后注定会过时淘汰；但你围绕业务建立起来的评测基准、质量门槛与决策框架，却能历久弥新。
Source quotes
[0:04] Udit Luther: beautifully wrong, but still so confident
[6:13] Udit Luther: if all that you have is a hammer, everything looks like a nail.
[11:04] Udit Luther: doing dumb things with LLMs
[11:04] Udit Luther: somewhere in this house.
[16:10] Udit Luther: Recognition is not equal to generation
[35:56] Udit Luther: Data prep is metered
[35:56] Udit Luther: Bad data isn't patchable
[35:56] Udit Luther: The spotter walks off
Original transcript
[0:04] Thank you for coming. My name is Udit Udit Luther.
Um My leader Jodie, she was supposed to be here,
[0:13] but for circumstances that are not in
my control, you're stuck with me. Um so,
[0:18] once again, thank you for joining. Um I lead
GenAI evaluations as well as model management at
[0:26] Cox Automotive from an AI/ML perspective. Um and
today we're going to talk about three things that
[0:33] we internally have been thinking about AI from.
Um so let's start with the problems first. Um
[0:43] You know, just by nods or through hands, please
let me know if in the past few months you've
[0:50] looked at your AI bill and your eyes have
popped out. Okay, so that's problem one.
[0:58] Okay, second. Um have you experienced AI be
beautifully wrong, but still so confident in
[1:06] whatever you've asked for that it doesn't really
seem like it's wrong. Okay, so that's problem two.
[1:16] And problem three, as we've been, you know,
working with AI and we've gotten used to it,
[1:23] who has suddenly started to feel like that it
should be faster? I should not be waiting like
[1:28] 10 seconds for a response now. So, that is problem
three. Um so, these are the problems that we will
[1:36] be talking about and how we have been internally
thinking about these problems and what are some
[1:42] of the ways that we've been working to kind
of eliminate [snorts] some of these problems.
[1:50] Okay, so cost first because money matters. Um from
an agentic perspective, right? Like agents are
[1:59] tuned to do their task no matter what. They don't
have this idea about what is expensive, what is
[2:06] not, and even if they do, maybe they do not care.
What what they're after is to complete the goal.
[2:15] And so, in doing that, they can run in circles,
they can read the same context over and over
[2:21] again, they can have tool usage that is not
really relevant to the task at hand anyway,
[2:29] but they will continue to do that until they
come up with an answer. So, the cost actually is
[2:35] multiplying at that point, right? Um but the cost
is not just for inference, right? Um if you are
[2:43] putting agents out there, you have cost related to
hosting them, their infrastructure, you have cost
[2:49] related to observability, you have cost related
to evaluations, you have cost related to security,
[2:54] etc. etc. So, the cost is actually increasing
just because you want to host an agent.
[3:02] So, [snorts] for people who have teenagers,
have you ever given your teenager a credit card?
[3:11] And if you have, does that make you nervous? I
mean, I'm assuming it would, right? And I feel our
[3:19] CFO kind of feels the same way. And it's it's not
that the issue is not that it's just expensive,
[3:27] the issue is also the variance. It could be
40,000 to 4 million. And so, the CFO cannot really
[3:35] plan for that, right? Like you
cannot really plan for teenagers. Um
[3:39] the saving grace may be that the agents at least
can complete the task, your teenagers might not.
[3:47] Okay, so the second one is uh I have this graph
on on the screen here, here, right? Like so,
[3:56] there are two axes. One is hard for humans, one
is hard for LLMs, and there is no real pattern
[4:04] there, right? So, things that are extremely hard
for humans might actually be very, very trivial
[4:12] for the LLMs themselves. However, for things that
are easy for us humans, these LLMs might trip over
[4:21] them. Right? So, it could be that um you know,
at our company, they've been able to provide,
[4:32] you know, great essays or great paragraphs
or great math on depreciation curves,
[4:38] but on the same kind of way and they've they've
put a V8 engine in a four-cylinder Camry as well.
[4:45] Right? And we built agents that
can actually help you, um you know,
[4:52] schedule an appointment with a dealership if
you're trying to buy a car on Friday, August 5th.
[4:58] And if you haven't caught that, Friday, August 5th
is not right, right? Like August 5th is tomorrow,
[5:05] but it would very confidently say that
I've booked you an appointment on Friday.
[5:10] So, we for the record, we've we've solved [snorts]
for that. But by default, it has done that to us.
[5:18] The second piece um of of this problem
is that you know, it's easy to just say
[5:25] we should evaluate our agents or we should,
you know, measure them or things like that, but
[5:30] ground truth is actually very hard to come by.
Especially given that all of these agents are
[5:37] mostly open-ended text, right? So, what what
is ground truth if I ask a question of like,
[5:43] "Give me suggestions of cars that are
family-friendly. If it returns a Porsche,
[5:51] is it the wrong answer? Debatable, right? So,
you know, just one thing, if nothing else, that I
[6:00] would want you to take away from this presentation
is that benchmarking and evaluations are actually
[6:06] pretty important. And to get there, we have to
come up with the ways to kind of measure stuff.
[6:13] Okay. So, the uh the final problem that we
talked about was latency. Right? So, latency
[6:21] is always thought about at the very end, but it's
the first thing that people would complain about.
[6:28] Um so, this this picture is about uh technician
that works on trucks. And like Cox owns um
[6:37] lead services as well. So, for this technician,
right? Like he's he's standing on the side of the
[6:43] highway, and traffic is zooming by him at like
70 mph. Um and you know, he's just waiting for
[6:51] an estimate to come to him so that he can get the
approval for the repair cost and start, you know,
[6:57] churning the wrenches. For him, it's not really
that he's thinking, "I should get this reasonable,
[7:06] very well-thought-out response from this LLM,
and that is the only time I will be happy."
[7:12] What he wants is that he gets an answers answer
as quick as possible so that he doesn't have to
[7:19] wait a minute more. He can get to his job, he can
make the repairs, and get out from the highway.
[7:25] So, the point being that, you know, you have
to think about latency beforehand and not
[7:34] after you kind of ship the product. For him,
if there were if it was taking like 10 seconds
[7:40] for every turn of conversation, he would actually
just not use the product and actually call someone
[7:47] who would give him what he needs. Um Okay. So,
so why do we have all these problems? And I think
[7:57] why that is the case is that if you all that you
have is a hammer, everything looks like a nail.
[8:05] In this case, the hammer is the LLM or the
frontier models that we're so used to now,
[8:10] right? Like it's one line of code,
you know, you just write that code,
[8:14] it works straight off the bat. You can throw
anything at it and it doesn't matter. It will
[8:19] try to do its best to answer whatever question
um that, you know, you're trying to solve for.
[8:26] But that's [snorts] that's fair, but would
you would you be wanting to use an LLM to do
[8:33] a classification problem? Would you want to use
an LLM to strip out dates from like a string?
[8:40] Would you want to use an LLM to ask for weather?
I mean, I'm sure all of us have done this,
[8:46] but there are better ways to do it. You can look
at your phone and get the weather out. Right? Um
[8:52] So, there are tools. What you might need is a
bigger toolbox, and that toolbox could contain
[8:58] smaller models, APIs, [snorts] um you know,
regular code written by people who you know, have
[9:05] written codes for decades, and once you've you've
written that code, it works well without any cost.
[9:11] So, it's it's things like that that we kind of
need to think about. So, let's let's talk about
[9:15] that. Um For me, what when I think of the toolbox,
there are three major things that come to mind.
[9:24] One is the agentic design principles, which
is basically to say, how do you think about
[9:31] building agents so that, you know, you don't run
into the problems that we we about. The second
[9:37] is model optionality, right? Like sure, we can we
can think about the frontier models, but there are
[9:43] other models also and and we'll talk about uh the
different models. I do want to make one note here.
[9:49] Um it it says pick one or build one. Um and
most of the talk is related to picking a model,
[9:58] but I do have some slides at the very end where we
will talk about is it worth making your own model
[10:06] and what should you think about if you do go that
route. And then finally, the third thing is the
[10:13] infrastructure, right? Like it not only
matters what model you're using, but it also
[10:19] matters where are you serving them, how are
you serving them. Um one thing that underlies
[10:25] all three and like I mentioned before is that you
wouldn't be able to kind of look at these or use
[10:32] them unless you have a way of measuring these
things. Right? Um if you don't have a way of
[10:38] measuring, there is no way you can know um whether
whatever changes you've made have actually helped
[10:45] um your system or not. Um so, that's not a
tool per se, but if you think about a toolbox,
[10:53] you know, if you don't have a tape measure,
you might have all the tools in the world,
[10:57] but without really measuring, the tools are
kind of useless. Okay, so let's let's go into
[11:04] the agenda principles first. Um I'm reiterating
here that benchmarking comes first. Right? I just
[11:13] said that if you cannot measure it, you don't know
what to improve or whether it has improved or not.
[11:20] Um so, data science as a practice has been doing
that for more than a decade now, right? Like
[11:26] uh you shouldn't be training your model on the
same data that you're kind of it in. That's just
[11:34] kind of cheating. Um and then benchmarking, uh you
know, creating the word, being on trend here, but
[11:42] benchmarking is more about like don't
try and follow one number, right? Like if
[11:48] all you care about is one number, you and the
model will find ways to increase that number.
[11:56] And if the number is only going in one direction,
sure, your metrics that you're kind of looking at
[12:02] might look good, but it might not actually be
helping the model itself or helping the system
[12:08] to be better to achieve what you're what you've
kind of set out for the system to achieve.
[12:14] Um the second one is don't do dumb things with
LLMs, right? So, we just talked about looking
[12:20] at weather. Um you can look at your phone
and you can get weather for free versus um
[12:28] you know, internally we've we've had like multiple
tools where, you know, we've cataloged vehicles,
[12:35] we have APIs that can give in a win tell you
exactly what the make, model, year, trim,
[12:41] etc., etc. of cars are, but we still have people
who would ask the LLMs what is the make, model,
[12:49] what features does this car have, etc. Sure, I
mean, the one like the model can give you some
[12:55] answer and it might be right, but but the point is
you're spending money for something that we could
[13:00] have just used our own API for. Um and the third
one is design for low cost and speed. And so,
[13:09] this the third point is more about intentionality,
right? So, maybe you think about the fact that
[13:15] there is a small classifier that you've put in,
you know, in front of your final query to say
[13:22] uh if this query is not as complex, I'm not going
to send it to my frontier model like an Opus,
[13:29] but I would send it to something like a Haiku. It
still does the job. And so um that just helps you
[13:37] to think about things of again, like using the
right tool for the right job. And so that would
[13:43] help um drive costs down as well as address
other uh problems that we had talked about.
[13:51] So um you know, [snorts] benchmarking
while it's easy to just say
[13:57] that go benchmark, but we talked about having
ground truth, right? So this is genuinely hard.
[14:04] Um and so therefore I do want to take a deeper
dive into how we've thought about benchmarking
[14:10] and how we have gone about um building these
benchmarks for not only our products, but also for
[14:17] trying to measure models across like open-source,
open-weight models um with the common clock
[14:23] standard so that we're comparing apples to
apples. So let's let's start with a simple
[14:29] kind of evaluation. Um so if you just imagine
that um you know, a car has come into a dealership
[14:38] and the engine light is on and you've uh had
some initial diagnostic codes that have been run
[14:46] and those diagnostic codes have been fed to
an LLM. So the LLM kind of responds with the
[14:52] fact that um you know, there's either a fuel
system leak or there is a sensor at fault. Um
[15:00] that answer is not wrong, right? It's
actually looking at the right system,
[15:05] it's in the vicinity but the answer is
more kind of equivalent to saying that
[15:12] the thing that you're looking for is
somewhere in this house or in this room.
[15:17] What the actual answer that was in this case the
root cause was the fact that the gas cap of the
[15:27] car was actually loose. Right? So, if you would
if you would if you think about this, right? So,
[15:37] not that the first one was wrong, but the second
one is actually more helpful. Right? The first one
[15:43] would have made the technician still
go around and look for the problem,
[15:48] while the second one would would fix the problem
immediately. And so, if we go back to that
[15:54] technician on the highway, what do you think
he would prefer? I'm I'm pretty sure he would
[15:59] prefer that he's just told the exact answer
without really trying to go through turns.
[16:10] The other issue with um you know, thinking about
benchmarks is uh you know, the these four things
[16:19] that we kind of discovered when we were trying to
um you know, dabble with training our own models
[16:25] and stuff. So, on the top left, recognition
is not equal to generation. Right? Like, if
[16:33] for example, your use case, the way you're testing
the model is through multiple-choice questions,
[16:41] picking the right answer from a multiple-choice
question is not the same as knowing the right
[16:46] answer. The LLM could actually eliminate choices
and get to the right answer without actually
[16:53] knowing the right answer. So, for systems
that broadly are in today's world open-ended
[17:01] systems, right? Like, checking accuracy through
multiple-choice questions might not be the right
[17:06] strategy. And so, you have to think about
how would you actually test the knowledge
[17:11] and not the ability to eliminate. The second
on the top right is self-recognition artifacts.
[17:19] And by that I mean that again, don't use
the same model to test and make the data uh
[17:28] which means that if if you've used the model to
train your agent and you've also used the model to
[17:34] create the data for testing, and you're using the
same model again to test it, the models themselves
[17:40] have at least in our experience are pretty good at
picking up things, right? So, Cloud generally for
[17:46] in our cases is very good at picking clouds own
work. So, you you have to be conscious of the fact
[17:53] that you're not inherently introdu- introducing
some bias that you're not actually aware of.
[18:00] The bottom left is the train and test format
mismatch. And by that I mean that if you've
[18:07] given [snorts] information to the model in
a certain format, in a certain way, in a
[18:11] certain phrasing, um when you when you're actually
trying to train the model, [snorts] and my sorry.
[18:22] Uh and you you know, you've given that information
to train the model, but in testing, you you're
[18:27] kind of presenting the same information in a much
different way that the model has never really
[18:33] thought of or come across that way, we found
that um you know, the model actually struggles.
[18:40] It's not that the knowledge is not there,
it's just that it doesn't understand um
[18:45] that the way that that has been presented is just
another way of presenting the same information.
[18:50] So, that's something that we've we kind of
realized after we were doing all this work.
[18:55] And then finally, contaminated ground truth,
right? Like um the first kind of thought you have
[19:02] when you say, "Okay, we need ground truth." and
you're like, "Okay, let's let's have some humans
[19:07] label some data." That's fair. I mean, that's that
is the way to do it, but recognize the fact that
[19:14] humans themselves could be wrong. Um in our
case, when we were trying to do an audit of
[19:20] the labels themselves, 35% of what we thought
were correct answers were them themselves
[19:26] defective. And And there's there's a human kind
of issue here as well, right? Like you have to
[19:33] be conscious of the fact that there is might be
a conflict of interest between the human who's
[19:39] trying to label the data for you uh and might not
want the system to succeed itself. So, you you
[19:44] have to be conscious of how you collect the data
and how you're kind of labeling the data itself.
[19:51] So, talking about humans, right? Um you know, at
the scale that we're kind of talking about, like
[19:59] humans are not going to sit there and evaluate
all systems. Um we've kind of narrowed it down to
[20:05] saying that um you know, we should probably use
a LLM as a judge, and that is a more effective,
[20:13] scalable way of figuring out whether
things are actually correct or not.
[20:19] But the graph on the left-hand side is an
out-of-the-box model trying to take a stab
[20:26] at whether uh you know, the responses coming
from another LLM are correct or not. So,
[20:32] the x-axis is what the human said, and the y-axis
is the LLM itself. Um it's pretty [snorts] diverse
[20:39] on the cloud, and then uh you From that graph, you
can't really tell who is wrong. Is the LLM wrong,
[20:46] or is the human wrong? And so, using judges out
of the box, at least in our experience, has not
[20:54] led to good results. And if you think about it,
right? Like if this was a human instead of an LLM,
[21:01] um what you're essentially doing is someone
who has knowledge, but has no training, has no
[21:09] um kind of concept of the realities of the job
itself, and you are putting this newbie from
[21:17] nowhere into this situation where they are
responsible for the quality themselves, right? So,
[21:23] So, like you you've not put them in
a in a position of success. And so,
[21:29] the graph on the right is what we kind of suggest,
which is make sure that when when you are thinking
[21:37] about trying to use LLM as a judge, you have data
that it can calibrate against. So, it's not that
[21:44] it's agreeing to you or it's agreeing to somebody
else, it's agreeing to the SMEs that you've vetted
[21:49] and you've um you're tasked with giving the right
answer. And so, as you do that more and more,
[21:56] uh you would expect that the alignment starts
to occur. And at that point, you would want to
[22:01] pin it, right? At least for a certain set of
experiments. You would want that you have used
[22:07] the same uh trained LLM as a judge uh to
make the comparisons fair and, you know,
[22:13] them being apples to apples. However, as the you
know, your agentic system evolves or has more
[22:22] kind of exposure to different tasks, the the
exercise would have to be repeated, right? Like,
[22:30] it's now judging more things, more diverse things,
and so you would have to repeat the same exercise
[22:36] to make sure there's coverage and it's
it's performing well in every scenario.
[22:44] Okay, so that was kind of a lot
on the agentic design principle.
[22:50] Let's talk about model optionality itself. Right?
So, I'm I'm I'll try and go from left to right.
[22:58] Um on the left-hand side, there is the
frontier model, right? Like you're paying
[23:02] dollars per millions Uh tokens. They they're
pretty slow. They're not available offline.
[23:09] You need an internet connection to
kind of go do that and to be fair,
[23:15] you need these models in tasks that are pretty
complex and need a lot of reasoning ability.
[23:22] The middle one SLMs on the other hand are about
10 times cheaper than the frontier models.
[23:29] They're relatively faster as well. They
could run offline but broadly possibly
[23:37] they they can run at least in low bandwidth
scenarios. However, they are more domain specific
[23:45] and they should be used for more specific and
narrower tasks. The right hand side is actually
[23:54] using edge or native mobile LLMs because
they're native the cost is broadly free.
[24:03] They are pretty fast and they can operate with
the with no internet and low bandwidth as well.
[24:10] But they're they're probably best suited
for predictable well-defined tasks.
[24:16] I have deliberately left two rows while I was
going through that. So let's talk about accuracy
[24:23] first, right? So accuracy is slightly surprising
initially at least, right? So LLMs at least in our
[24:31] kind of testing and our work they're good,
right? Like they can score about 70 to 85% but
[24:39] when you fine-tuned an SLM we at least have
seen that the accuracy is better than the
[24:45] frontier model because you've given them given
the models relevant data to kind of look at and
[24:52] you know reason on. So we've we've seen
the accuracy to go up to about 85% to 95%
[24:59] I am going to give you a caveat
here though, right? Like we have
[25:05] the infrastructure and the engineering
resources and model up resources kind of do
[25:12] all of this experimentation you know
in your scenario that might not be the
[25:18] case or it might be the case. But if it's not
the case, right? Like you have to think about
[25:23] cost from that [snorts] perspective as well.
Like getting model up steam set up or engineering
[25:29] resources set up on infrastructure set up that
has to be part of [snorts] the equation and then
[25:36] if that is happening maybe the frontier
model is the right choice [snorts] to go
[25:41] with and that serves the purpose without a
lot of kind of investment from your side.
[25:49] Okay, [snorts] and so after models there
is the serving of the models you know,
[25:56] where should the models be for you to get the
inference? I think this slide is anchored by the
[26:04] by the bar below, right? So what that bar
is basically saying is from left to right
[26:13] you have lower volume of transactions, there's
lower commitment and you only pay for what you use
[26:21] and on the right hand side you know, the volume
that you need kind of service is pretty high,
[26:27] there's a high fixed cost and even if you don't
use things you will still be kind of built for it.
[26:33] So that's that's the spectrum that we're going to
talk about. So let's start on the left hand side
[26:40] which is edge or on device. Right?
So runs where the work happens,
[26:46] there is no per call cost and the data is not kind
of leaving your domain. It's generally good for
[26:53] tasks that are small and that you can repeat.
The next one is on-demand [snorts] API.
[27:00] And in our case, we we are an AWS shop heavily.
And so, a lot of this analysis that is being
[27:06] shown to you is kind of um, anchored around our
experience with the AWS services. And so, um,
[27:13] you know, the on-demand API was Bedrock managed.
Uh, you kind of pay per token. Um, and you do not
[27:20] pay anything when this when the infrastructure
is idle. Um, it's it's best for use cases where
[27:28] you know, you want to do or execute batch jobs
or your traffic is actually bursty and not,
[27:34] um, you know, a continuous stream of data. Um,
the third one is the custom model import, which
[27:41] basically means that you can train your own
model or you can bring a model uh, directly
[27:46] from the open source world or the open weight
world. And you can put that into Bedrock. Um,
[27:52] Bedrock then bills bills you by what they call
capacity unit windows, which is basically they are
[27:59] charging you by a bucket of infrastructure that
they've kind of already made for you. And they
[28:06] charge you by the minute for that bucket. Um, this
is the cheapest past path [snorts] to a customized
[28:13] open model if you really need it. And then
finally, if there is a provisioned endpoint way of
[28:19] doing things, which basically means, just like the
frontier models, you also have your own API now.
[28:25] Um, and so, it is dedicated to your service 24/7.
Um, you obviously have a lot of control and you
[28:31] can set it up however you want. Um, but the fact
is that you have to pay for it all the time,
[28:36] even if your service is not being utilized,
you still pay the infrastructure cost.
[28:41] So, these are these are things that you
just have to kind of keep in mind if you
[28:47] think about going down the route of having your
own models and serving inference as well. Um
[28:55] my final kind of thought on um you know, just
the agentic principles is something I want to
[29:01] close out um based on cost itself. Um within
the cost domain, these three verbs are actually
[29:11] used pretty often and sometimes they're
confused. So, I just wanted to kind of throw this
[29:16] um onto the slides to say um there are
three ways to actually control cost.
[29:22] Um and actually the first two are the only
ones that would make a dent in you can trying
[29:27] to control your cost. Third one is just um it's
just a descriptive analysis of what has happened
[29:34] um with with your uh system. So, govern is
that uh we're we're trying to think about um
[29:42] you know, making sure that not all teams
have access to all models um by use case, by
[29:50] um what you want to achieve. Maybe you are
only limited to a certain set of models.
[29:57] And if it's only that you're doing this
cutting-edge work, that cutting-edge models
[30:02] are made available to you. But, there are other
concerns that also go into um this point because
[30:09] at least within Cockroach, right? Like we've we've
thought about security concerns, we've thought
[30:13] about um engineering constraints, we've thought
about um the quality of the models, etc. So,
[30:23] govern is a catch-all kind of uh bucket for
you to think about. The second is the route,
[30:31] which is the right model for the right job.
Like I said, um if we just want a simple answer,
[30:38] we don't probably need to go to Opus. We can uh
route to uh you know, one of the haikus. Um But,
[30:45] what we've realized is that this is more than just
education. Like, if you leave people to their own
[30:53] um kind of um you know, decision, they'll they'll
probably not take that decision. They would
[31:00] probably think that you I want to give everything
to Opus. So, what we're trying to think about or
[31:06] at least search for is automated intelligent
routing of queries so that um you know,
[31:12] you do not have the burden of making the choice.
There is a system that makes the choice for you.
[31:19] And and gives you gives the request
to the right model at the right time.
[31:24] And then finally, account, right? Like,
account is like you you go in and see
[31:28] what has happened. Um did we where did we
lose a lot of money? Where did we try and do
[31:36] um spends that were not uh necessary? Things like
that. But, like I mentioned before, right? Like,
[31:42] it can only tell you what has happened.
It cannot control um uh your cost. And so,
[31:50] it's better to be proactive about things than
than just relying on the accounting part of it.
[31:57] Okay. So, um you know, we're kind of coming
towards the end of the presentation. I had
[32:03] talked about um you know, what to think
about if you really want to go down
[32:10] this route of making you or tuning or
serving your own models, right? Um so,
[32:16] there are a few considerations that I would like
to point out before you actually try and go into
[32:23] um that route. Um so, first of all, I think
the gate that you need to think about is
[32:30] do you actually need to customize anything,
right? Um whatever your use case is, is it
[32:35] not being served by whatever is available off the
shelf? And [snorts] if you do want to customize,
[32:43] the things that you have to think about
are do you have the right data, right? Um
[32:50] What what is it that you're trying to achieve
that these frontier models have not already,
[32:56] um you know, consumed in terms of knowledge?
And so, if the answer is there's nothing new,
[33:03] then there's probably not a you know, a case
for for going down this route. In our case,
[33:10] right? Like for example, we have data that
talks about uh service and repair for cars,
[33:17] right? Because we have dealerships um that use
our software. And so, we have tons of transactions
[33:23] that can tell uh you know, what customers own
what cars, what kind of uh repairs they made,
[33:30] uh at what time, how much did that cost. So,
there's a lot of information there of like owning
[33:37] the car itself and how much do you spend on a car
and what does it cost, etc. etc. That information
[33:43] is not publicly available information. And so,
it's important that uh we use that information
[33:49] to [snorts] kind of guide the models to give
us answers that are relevant to our questions.
[33:56] The second gate is um back to benchmarking. Um you
know, you you have to kind of do the hard work. Um
[34:05] you have to for your given use case,
create benchmarks, um put in the time,
[34:12] uh make sure that it covers a wide variety
of use cases within um whatever project that
[34:18] you're trying to do. Because if you again,
if you don't have this, there is no way you
[34:23] can measure and there's therefore there is no way
you can know if you've actually improved or not.
[34:29] The third gate, if you've passed the first
two, is have you eliminated um you know,
[34:37] techniques that have been talked about for
the last few years, right? Like and that
[34:40] includes prompting, it includes retrieval, um
it also could just mean using a smaller model.
[34:48] Um so, it's things like that and it's only if
you think that you're able to eliminate all of
[34:55] these techniques and say these techniques are not
really uh [snorts] doing what I need them to do
[35:01] is when you say, "Okay, now I think I I should
move forward." And so, the final gate that you
[35:07] kind of need to think about is can you afford to
serve the thing once you've built it, right? So,
[35:14] serving is not just cost, right? Like you
obviously have to think about cost, right? Like do
[35:20] you need GPUs? What infrastructure do you need? Um
but it's also a lot of other things that I'm going
[35:27] to talk about on the next slide as well, but you
have to manage security, you have to manage the
[35:31] infrastructure yourself. A lot of things that we
get from these frontier model companies in terms
[35:36] of optimizations that they've done to serve these
models, like you lose all of that. And so, this
[35:44] question needs to actually be answered before you
start training a model rather than after you've
[35:49] trained the model and then you're trying to figure
out, "Okay, can we actually serve this model?"
[35:56] So, like I mentioned, right? Like when you
do take on customization, you you do miss
[36:04] out on a lot of things and so, let's start with
things on the right-hand side. Um the first one,
[36:11] it says that the prep is metered. What that
means is that every single turn that you take
[36:17] to kind [snorts] of curate your data for training
this model is on you and every turn is a bill,
[36:23] which means that um whatever change you're
making is going to cost you and you have to
[36:31] think about that cost as you kind of go down this
path of curating the data for training the model.
[36:37] Um the second one is something we experienced
ourselves. Um bad data isn't patchable. What
[36:45] that means is that for example, if the training
data for some reason had leaked PII information,
[36:53] there is no way for you to undo that, right?
Like you have to remove the data from the
[37:00] PII data from the data set and then train the
model all over again, which means that all
[37:06] everything that you spent initially in trying
to train the model is gone down the drain.
[37:12] The third one is the spotter walks off, which
which is referring to what I was mentioning,
[37:17] right? Like these these frontier model
companies have been doing a lot of things
[37:22] um to keep you safe, right? Like safety has been
trained into the weights of these models um from
[37:28] the start. So, if you take on the training of
models, you have to think about the fact that your
[37:35] models need to be safe, your models need to not
be toxic and things like that. And then finally,
[37:43] um you know, you you could be working on
different aspects of training the model
[37:49] and you might fix one aspect in
this one place on the side here,
[37:53] but it might have like domino effects on
things in another part of the training
[37:58] and it might mess that up. And so, it's you are
responsible for that and you have to be aware
[38:05] of uh things like this that can happen and that
can throw off your entire training process, which
[38:12] means coming back like measuring and evaluations
are actually an important part of your process as
[38:19] well and you have to think about those things
from the very beginning and make sure you have
[38:23] benchmarks or make sure you have evaluations that
as and when you keep trying to train these models,
[38:30] you know that nothing is actually going wrong.
So, the grid on the left-hand side, I think does
[38:35] a good job of visually kind of explaining what you
own, right? So, as input gets less structured and
[38:43] as output gets less structured, there is more for
you to own. Um and depending on where you stand,
[38:51] it might or it might make sense or it might
not make sense for you to kind of get into this
[38:56] business of training the models. Okay, so finally
to kind of wrap up, right? Um cost, correctness,
[39:03] and latency were the things we identified as
problems. Um just from a cost perspective,
[39:10] make sure that you govern um you know, route
the non-complex things to smaller models and
[39:16] not necessarily frontier models. Keep the frontier
models reserved for uh complex problems. And then,
[39:23] train or build models if absolutely necessary
and make sure you think about the economics
[39:29] of serving the models. On the correctness side,
um make sure you define what is right, right? So,
[39:37] um are you giving the answer of the system or
are you giving the answer of the part itself?
[39:41] What What does right constitute for you? Uh you
have to kind of define that. Build the benchmarks,
[39:47] calibrate the judges, and do it for all use
cases that you have, not just one or two.
[39:54] And then finally, pick the serving
path for latency, right? On demand
[39:59] um or sorry, on device might be a real option
um for the right use case that you might have.
[40:04] Um if there's anything that you can go
do today, uh you can already uh you know,
[40:09] look at something that you've shipped and start
writing down what does good mean for that.
[40:14] And you can start to then build the process of
measuring from there if you haven't done that,
[40:19] right? So models in my opinion will be obsolete
whatever you use in a year, but the bar or or
[40:25] the framework that you build around it might
not be. So um that's it for me. Thank you.
