## Background
Published on YouTube by Ai4 on August 27, 2025, this panel discussion focuses on strategies for building robust computer vision data pipelines, centered on data labeling, dataset curation, and data management. Moderated by Saurabh Kumar and featuring industry practitioners from LinkedIn, 2K Games, iMerit, and Perle, the session outlines methodologies for handling large-scale datasets, synthetic data augmentation, and continuous learning in computer vision workflows.
## TL;DR
The panel explores the unique challenges of computer vision pipelines compared to standard NLP systems, emphasizing the criticality of physical ground truth, input pixel integrity, and lineage-backed promotion gates to prevent silent production drift. The speakers unpack the realistic trade-offs of synthetic data, noting its utility for constrained domain adaptation while cautioning that deliberate human-in-the-loop oversight is mandatory to avoid compounding distribution errors. They also examine production architectures for multimodal vision models, including embedding-space drift tracking, offline-to-online knowledge distillation to control GPU costs, and linguistic talk tracks for autonomous system explainability. Across all topics, the experts stress an empirical, hypothesis-driven approach to data engineering, reminding teams that continuous micro-iteration outweighs reliance on automated silver bullets.
## Atmosphere & style
The discussion carries a pragmatic, battle-tested tone shaped by experienced engineers and data operations leaders sharing lessons from the production trenches. Rather than indulging in ungrounded industry hype around generative AI and full automation, the panelists maintain a candid, skeptical perspective on synthetic panaceas and off-the-shelf playbooks. The interaction is collegial and fast-paced, offering viewers an authentic, unvarnished look at the tedious calibration, edge-case triage, and architectural compromises required to keep high-stakes computer vision pipelines running reliably.
## Key takeaways
- [01:48](https://www.youtube.com/watch?v=UZv87qrKM54&t=108s) Unlike NLP pipelines that deal largely with subjective or text-based data, computer vision systems map to physical reality, enabling mathematically rigorous ground truth and quality control standards.
- [04:02](https://www.youtube.com/watch?v=UZv87qrKM54&t=242s) Deep learning models can tolerate architectural noise, but corrupt pixel data and visual distribution drift will fundamentally break feature extraction and model performance.
- [04:02](https://www.youtube.com/watch?v=UZv87qrKM54&t=242s) Pipeline failures are often misattributed to annotator errors when the actual root causes stem from collection timing, data volume issues, or distribution mismatches.
- [06:26](https://www.youtube.com/watch?v=UZv87qrKM54&t=386s) Production deployments benefit from lineage-forced promotion gates, blocking any model deployment if unexpected performance anomalies cannot be traced back to a specific data slice.
- [10:05](https://www.youtube.com/watch?v=UZv87qrKM54&t=605s) Synthetic data is effective for bootstrapping prototypes and bypassing privacy barriers like HIPAA, but it fails to replicate unpredictable real-world edge cases in uncodified domains like medicine or law.
- [14:52](https://www.youtube.com/watch?v=UZv87qrKM54&t=892s) Rather than waiting for end-to-end training runs to evaluate data, pipelines should enforce modular verification and early warning quality audits at each intermediate stage.
- [22:20](https://www.youtube.com/watch?v=UZv87qrKM54&t=1340s) Combining vision systems with language models provides explainability through human-readable decision 'talk tracks,' making erratic autonomous edge cases easier to diagnose and debug.
- [26:06](https://www.youtube.com/watch?v=UZv87qrKM54&t=1566s) In the earliest stages of multimodal model development when weights are highly volatile, the cost of data and human annotation errors compounds logarithmically.
- [28:46](https://www.youtube.com/watch?v=UZv87qrKM54&t=1726s) Transitioning to prompt-driven multimodal foundation models introduces new failure modes, including extreme prompt sensitivity, opaque inter-model communication, and reduced explainability.
- [31:32](https://www.youtube.com/watch?v=UZv87qrKM54&t=1892s) To scale serving to high user volumes affordably, production teams use large frontier models offline for complex extraction and knowledge distillation to power lightweight online models.
- [38:21](https://www.youtube.com/watch?v=UZv87qrKM54&t=2301s) Detecting distribution drift in high-dimensional visual data requires projecting live production data and training baselines into embedding spaces to measure vector distance anomalies.
- [40:28](https://www.youtube.com/watch?v=UZv87qrKM54&t=2428s) When training exclusively on synthetic data, teams must drastically narrow the problem scope and fine-tune existing foundation models rather than training weights from scratch.
- [43:28](https://www.youtube.com/watch?v=UZv87qrKM54&t=2608s) Synthetic data pipelines are subject to temporal degradation, requiring continuous real-world edge-case testing and human-in-the-loop guidance rather than being treated as permanent silver bullets.
## Detailed walkthrough
### [00:04](https://www.youtube.com/watch?v=UZv87qrKM54&t=4s) Panelist Introductions and Backgrounds
The session opens with brief introductions from each of the panelists, highlighting their diverse paths across industry verticals and research domains in artificial intelligence:
* **Vijayraj Gohil**: Machine Learning Engineer at 2K Games, with a background spanning retail tech machine learning and ML research at New York University (NYU).
* **Teresa O'Neill**: Linguist and data operations specialist serving as Director of Solutions Architecture for Generative AI at iMerit, focusing on data services and AI data transformation across the automotive, medical, and enterprise generative AI sectors.
* **Arpita Vats**: Senior AI Engineer at LinkedIn, whose PhD research bridged computer vision and natural language processing, leading into three to four years of focused work in large language models (LLMs) and generative AI.
* **Ahmed Rashad**: Founder and CEO of Perle. Prior to founding Perle, he built and scaled human data operations at Scale AI from 2018 onward, engineered AI applications at Amazon and McKinsey, and studied operations research at MIT after starting his career as an offshore driller.
```mermaid
flowchart TD
accTitle: AI Panelist Backgrounds and Focus Areas
subgraph Panelists["Panel Expertise"]
V["Vijayraj Gohil<br/>(2K Games / NYU)"]
T["Teresa O'Neill<br/>(iMerit)"]
A["Arpita Vats<br/>(LinkedIn)"]
AR["Ahmed Rashad<br/>(Perle / ex-Scale AI)"]
end
V --> D1["Data Contracts & Lineage"]
T --> D2["Ground Truth & Annotation Ops"]
A --> D3["Multimodal CV & Data Quality"]
AR --> D4["Human-in-the-Loop & Root-Cause Ops"]
```
---
[01:48](https://www.youtube.com/watch?v=UZv87qrKM54&t=108s) What Sets Computer Vision Pipelines Apart: Multimodality and Ground Truth Rigor
Saurabh Kumar begins the discussion by asking the panel how computer vision (CV) pipelines diverge fundamentally from traditional tabular or NLP data pipelines. Arpita Vats points out that while NLP pipelines predominantly handle text or audio streams that can be transcribed and standardized into uniform text formats, vision pipelines are inherently multimodal and structurally complex. Computer vision pipelines must synchronize image arrays, video streams, audio channels, fine-grained timestamps, and subtitle metadata into a unified workflow. This high-dimensional complexity requires the ingest and validation pipelines to be substantially more rigorous than those designed for single-modality tasks.
Teresa O'Neill follows up with an essential philosophical distinction between generative NLP systems and vision systems. In computer vision, models target the physical, measurable real world where an absolute ground truth exists: objects occupy defined coordinates, follow physical laws, and exhibit knowable motion patterns. By contrast, large language model pipelines often attempt to capture cognitive, subjective, or social realities where verifiable ground truth cannot be directly measured. Because visual data corresponds to concrete physical phenomena, teams can implement objective, mathematically rigorous quality control standards and requirements design that would otherwise remain ambiguous in text-based workflows.
```mermaid
flowchart LR
accTitle: Comparison of Computer Vision vs. NLP Data Pipelines
subgraph CV["Computer Vision Pipeline"]
CV_In["Visual Inputs<br/>(Video, Audio, Subtitles, Timestamps)"] --> CV_GT["Physical Ground Truth<br/>(Knowable coordinates & physical laws)"]
CV_GT --> CV_QC["Objective & Rigorous Quality Control"]
end
[04:02](https://www.youtube.com/watch?v=UZv87qrKM54&t=242s) Failure-Prone Links, Quality Control, and Root Cause Attribution
When asked to identify the single most failure-prone phase within the computer vision lifecycle—whether ingestion, labeling, versioning, or monitoring—Arpita highlights raw input data quality and pixel integrity. While neural networks can tolerate a certain baseline of mathematical or architectural noise during optimization, corrupt pixel data, artifacts, and out-of-distribution imagery make feature extraction virtually impossible. Arpita also notes that continuous dataset versioning is critical; failing to update visual benchmarks causes the training distribution to drift severely from the real-world operational environments encountered during inference and production serving.
Ahmed Rashad expands on the problem by arguing that failures can occur at every node—ingestion, volumetric scaling, enrichment, or human annotation—but the primary operational bottleneck is root-cause attribution. Teams frequently misdiagnose model degradation by blaming annotation errors when the true issue lies in insufficient data volume, sampling from the wrong distribution, or ingesting data out of sequence. Because human labelers introduce natural variability, operations teams frequently jump to conclusions rather than systematically isolating whether the breakdown originated in requirements specification, data collection, or labeling execution.
- **Data Quality vs. Model Noise**: Deep learning models can absorb minor architectural or parameter noise, but corrupted pixel inputs degrade downstream learning entirely.
- **Continuous Distribution Alignment**: Dataset versioning must reflect current production distributions to prevent deployment failures.
- **Attribution Confusion**: Pipeline failures are frequently misattributed to poor labeling when collection bias or timing issues are the true root causes.
---
[06:26](https://www.youtube.com/watch?v=UZv87qrKM54&t=386s) Calibration Knobs: Data Contracts, Drift Monitoring, and Pre-Labeling vs. Human Intervention
Discussing the operational levers that balance turnaround SLAs, quality assurance, and unit costs, Ahmed warns against treating any single methodology—such as synthetic generation or pure automation—as an absolute silver bullet. Optimal pipeline design depends strictly on the model lifecycle phase:
1. **Golden / Benchmark Datasets**: Teams should rely heavily on expert human labelers with minimal machine pre-annotation to preserve ground truth integrity.
2. **Pre-Training & Mass Production**: Pipelines should shift toward machine-driven pre-labeling with light human-in-the-loop oversight to maximize scale and contain costs.
Ahmed emphasizes that the industry has not "fully cracked" the exact mathematical relationship between operational knobs and downstream model performance, requiring iterative calibration based on empirical results rather than dogma.
Vijayraj Gohil shares practical mitigation strategies for data labeling inconsistencies and schema evolution. In gaming and production environments, Vijayraj implements strict **data contracts** alongside **drift monitors** that track how model accuracy shifts across different training data batches. He enforces a **lineage-forced promotion gate**: if an unexpected performance shift cannot be traced directly back to a specific data slice and its corresponding lineage, the pipeline blocks the model from being promoted to production.
```mermaid
flowchart TD
accTitle: Lineage-Forced Model Promotion Architecture
A["Raw Ingest Data"] --> B["Data Contracts & Schema Validation"]
B --> C["Training Dataset Slices"]
C --> D["Model Training & Evaluation"]
D --> E{"Drift Monitor & Lineage Check"}
E -- "Traceable Lineage & Performance Verified" --> F["Promote to Production"]
E -- "Untraceable Anomaly / Drift Detected" --> G["Block Deployment & Isolate Slice"]
```
---
[09:11](https://www.youtube.com/watch?v=UZv87qrKM54&t=551s) Synthetic Data and Data Augmentation: Closing Gaps vs. Reality Gaps
Turning to data augmentation and synthetic generation, Saurabh asks the panelists to highlight where synthetic methods effectively close performance gaps versus where they fail due to long-tail edge cases or overfitting.
Teresa cautions that synthetic data is not a general "panacea" despite current industry hype. Successful synthetic generation requires a deliberate, hypothesis-driven design cycle that defines precisely what missing distribution gap the synthesized data is intended to bridge. She notes that in high-stakes environments—such as healthcare computer vision applications—synthetic assets can provide valuable signal and fill sparse classes, but they strictly require continuous human expert oversight to prevent the model from learning synthetic artifacts or widening the reality gap.
[10:05](https://www.youtube.com/watch?v=UZv87qrKM54&t=605s) Synthetic Data in Regulated Domains and the Importance of Deliberate Design
Teresa O'Neill highlights how synthetic data can bridge critical data availability gaps, citing an LLM healthcare project where real-world data collection was heavily constrained by privacy and HIPAA regulations. By synthetically generating simulated clinician-patient interactions across various medical subspecialties and multiple languages, the team produced an effective training dataset. While performance was solid, the initiative still required rigorous human monitoring. Teresa emphasizes that synthetic data is most effective when applied deliberately to address explicit gaps in the data lifecycle, allowing teams to balance accuracy, safety, and cost without taking unnecessary risks. Echoing Ahmed Rashad's earlier point, she describes model development as "turning the right knobs at the right time."
Ahmed seconds Teresa's perspective, emphasizing that synthetic data is useful for quickly bootstrapping a proof of concept (PoC), but falls short because it generates predictable patterns rather than capturing true real-world complexity. In specialized domains such as medicine and law—where the entirety of human knowledge has not yet been codified or ingested by AI—synthetic generation often manufactures artificial edge cases that have no grounding in reality while completely missing genuine, high-stakes edge cases. When even domain experts like doctors hold divergent opinions on a diagnosis, synthetic data cannot replicate that ambiguity because the underlying knowledge has not been digitized or ingested.
```mermaid
flowchart TD
accTitle: Trade-offs of Synthetic Data in Specialized Domains
subgraph GapAnalysis["Data Availability & Privacy Strategy"]
A["Real-World Data Constraints<br/>(HIPAA, Privacy, Access)"] --> B["Deliberate Synthetic Data Generation<br/>(Clinical & Multilingual Dialogues)"]
end
subgraph EvaluationLoop["Human Oversight & Ground Truth"]
B --> C["Rapid Prototyping / PoC"]
B --> D{"Known vs. Unknown Knowledge"}
D -->|"Predictable Data"| E["Human-in-the-Loop Monitoring"]
D -->|"Expert Disagreement / Edge Cases"| F["Requires Real-World Ground Truth"]
end
```
- Synthetic dialogue was successfully used to simulate doctor-patient conversations across diverse medical subspecialties and languages.
- Synthetic generation reduces regulatory risks around HIPAA compliance while keeping failure stakes manageable through active oversight.
- In domains with uningested or contested human knowledge, synthetic models produce predetermined answers that fail to reflect actual real-world ambiguities.
---
[12:27](https://www.youtube.com/watch?v=UZv87qrKM54&t=747s) Data Lineage, Scientific Iteration, and Root Cause Debugging
Arpita Vats directs the conversation toward data pipeline infrastructure, specifically asking the panel how they handle data versioning, lineage contracts, and schema bugs. She notes that when a computer vision pipeline breaks or a schema bug is discovered, engineering teams must be able to instantly reconstruct which model ingested which slice of labeled data at any given timestamp.
Ahmed responds by warning against mass data labeling and cleaning just for the sake of following standard "best practices," arguing that builders creating novel technologies cannot rely on generic playbooks. Instead, engineers must adopt a rigorous scientific method: form a clear hypothesis, execute the task, and evaluate whether the actual outcome matches expectations. When discrepancies appear, teams must immediately halt the pipeline to isolate whether the failure stems from the machine, human annotators, or the underlying process. Ahmed stresses that rapid daily iteration—making hundreds or thousands of micro-tweaks—is far more critical to long-term success than pushing through massive volumes of unvalidated data.
- Traditional "best practices" often do not apply when engineering novel AI systems; builders must rely on structured hypothesis testing.
- When an unexpected output occurs, engineers must determine whether the root cause lies in the machine infrastructure, the human labelers, or the pipeline process before proceeding.
- Iterating daily with micro-adjustments prevents teams from accumulating compounding technical debt compared to slower, weekly review cycles.
---
[14:52](https://www.youtube.com/watch?v=UZv87qrKM54&t=892s) Phase-by-Phase Pipeline Verification and Early Warning Quality Audits
The panel debates the optimal cadence for pipeline validation. While acknowledging the value of backtracking, panelists emphasize that engineers should not wait for an entire end-to-end pipeline to finish before inspecting results. Instead, each pipeline stage should be validated against its own explicit output contract and distribution expectations. Verifying intermediate representations step-by-step may extend project timelines from one week to two, but it prevents costly downstream debugging across the entire pipeline. To detect data corruption early, teams can evaluate the underlying assumptions of automation scripts and embed validation questions within human annotation workflows to gauge annotator alignment.
```mermaid
flowchart LR
accTitle: Modular Computer Vision Pipeline and Quality Auditing
subgraph Ingestion["Stage 1: Raw Ingestion"]
A["Source Data Collection"] --> B["Triage & Indexing"]
end
subgraph Validation["Stage 2: Operational Oversight"]
B --> C{"Labeling Ops Early Warning"}
C -->|"Distribution Shift / Anomalies"| D["Immediate Batch Quarantine"]
C -->|"Passed Inspection"| E["Annotation & Schema Audit"]
end
subgraph Training["Stage 3: Downstream Execution"]
E --> F["Pre-training & RL Pipelines"]
F --> G["Production Deployment"]
end
```
Teresa expands on this by framing data pipeline management through the lens of traditional manufacturing quality control and agile R&D. Partnering with a dedicated labeling operations team provides an essential early warning system before raw data ever reaches compute-intensive model training. She shares real-world examples where source data contained unforeseen anomalies—such as autonomous driving footage captured during local cultural festivals or extreme flood conditions where motorists drove erratically. Because raw data was indexed across batches, human annotators flagged these distribution shifts immediately upon ingestion. Establishing robust, independent auditing programs for both raw inputs and annotations allows teams to fix schema drift and data corruptions early, avoiding wasted compute and post-deployment model drift.
- Modular data verification requires validating intermediate data distributions rather than debugging only after full model evaluation.
- Annotation pipelines benefit from embedding targeted validation checks to identify human annotator bias and flawed tooling assumptions.
- Batch indexing and operational partnerships surface real-world edge cases (e.g., severe storm flooding, festival traffic patterns) before expensive GPU pre-training begins.
- Independent quality audits across raw and annotated data prevent distribution drift from reaching production environments.
---
[18:56](https://www.youtube.com/watch?v=UZv87qrKM54&t=1136s) Transitioning to Real-World Computer Vision Applications
Arpita transitions the discussion from operational frameworks to specific industrial applications, inviting the panelists to share practical examples of computer vision pipelines they have designed and the core problems those systems were built to solve. The engineers begin reflecting on their hands-on work across gaming, autonomous navigation, and specialized production environments.
[19:26](https://www.youtube.com/watch?v=UZv87qrKM54&t=1166s) Retail Checkout Noise Removal and Industry vs. Research Computer Vision
Addressing the challenges of developing automated retail self-checkout systems—similar in concept to Amazon Go where customers select items without a cashier—a major hurdle occurs at the checkout station when customers set down multiple items at once. Surrounding clutter and occlusions, particularly human hands holding or scanning items, create substantial visual noise that makes it difficult for camera systems to isolate individual items. To solve this, the team developed an inpainting algorithm designed to strip away surrounding noise while preserving the underlying object's identity. Because the core models were trained on 2D images while the actual merchandise is three-dimensional, the training pipeline had to incorporate images captured from numerous angles so the network could accurately recognize the items based on contrast and geometry, even when parts were occluded.
Reflecting on the distinction between commercial deployments and foundational research, computer vision challenges generally split into two domains:
- **Industry deployments**: Teams focus heavily on infrastructure, processing high data volumes, fine-tuning and combining existing foundation models (such as chaining object detection with segmentation), online sampling during testing, and real-time model monitoring.
- **Research settings**: The focus shifts to raw capability improvements—investigating which specific datasets directly enhance foundational segmentation, detection power, and generalization.
---
[22:20](https://www.youtube.com/watch?v=UZv87qrKM54&t=1340s) Multimodal Explainability and Reasoning in Autonomous Systems
With computer vision roots anchored in autonomous mobility, industry focus is shifting toward multimodal architectures to address explainability and trust. While classical object detection and behavioral prediction models continue to mature, their failure modes remain notoriously erratic and opaque: "when the model fails, it fails so hard and so weird." To unpack this black box, developers are hybridizing perception systems with language and reasoning models. This allows autonomous systems to verbalize their real-time decision-making into an audible or readable "talk track" (e.g., stating that they observe a pedestrian who appears poised to jaywalk and are actively choosing to slow down).
subgraph OUTPUT["Action & Safety Loop"]
DEC["Vehicle Control Action"]
DIAG["Edge Case & Failure Diagnosis"]
TRACK --> DEC
TRACK --> DIAG
end
```
This human-readable narration provides critical diagnostic visibility when edge cases occur. For example, if a model misinterprets a reflection in a store window as a pedestrian, or encounters an unusual animal such as a peacock blocking traffic in Miami—where peacocks notoriously refuse to yield to cars unlike other wildlife—the generated rationale reveals precisely where perception diverged from reality. Engineers can then isolate the specific risk scenarios, execute targeted test simulations, and shore up the model via reinforcement learning.
---
[24:34](https://www.youtube.com/watch?v=UZv87qrKM54&t=1474s) Robotics and Contextual Multimodality
Beyond self-driving vehicles, drones, and delivery bots—domains where core autonomous navigation is increasingly mature—the frontier of computer vision has moved toward robots operating in complex, variable environments. In home assistance and industrial automation, tasks demand contextual judgment rather than simple pattern recognition.
A prime example is agricultural robotics, such as automated strawberry harvesting. A picking robot must assess the exact ripeness level of an individual fruit, decide whether to harvest and pack it, and ingest downstream feedback explaining why a particular pick was incorrect. Closing this learning loop requires pairing vision models with language models to interpret contextual nuances, marking multimodality as the essential path forward for next-generation robotics.
---
[26:06](https://www.youtube.com/watch?v=UZv87qrKM54&t=1566s) Early-Stage Failure Modes, Data Quality, and Human Annotation Sensitivity
When scaling multimodal systems that fuse vision, language, and tabular data into production, initial failure modes almost always trace back to foundational data collection and curation errors:
1. **Lack of the right data**: Models cannot generalize without representative source data.
2. **Insufficient volume of correct data**: Attempting to mask shortages by prematurely injecting synthetic data or mismatched external datasets can degrade accuracy. In the early stages of model development, when weights are highly volatile, "the cost of a failure is significantly higher" and errors compound on a logarithmic scale.
3. **Inconsistent human annotation**: Turning raw inputs into high-quality training assets relies heavily on continuous human-in-the-loop evaluation and labeling. Maintaining rigorous labeling standards across large workforces is inherently challenging due to human variance—subtle daily fluctuations in annotator precision can disproportionately corrupt an early-stage, highly sensitive model.
---
[28:20](https://www.youtube.com/watch?v=UZv87qrKM54&t=1700s) The Surge of LLMs in Computer Vision Research
The convergence of language and vision has become the dominant paradigm across premier research venues like CVPR (Conference on Computer Vision and Pattern Recognition). Research papers are overwhelmingly exploring large language models to contextualize visual scenes, interpret multi-agent environments, and guide decision pipelines for autonomous driving at scale. However, significant interpretability challenges remain, as the exact internal mechanics and representational shifts within these large language models are still not fully understood.
[28:46](https://www.youtube.com/watch?v=UZv87qrKM54&t=1726s) The Black-Box Challenge of Multimodal AI and Evolving Failure Modes
Multimodal AI models have made machine learning pipelines significantly easier to manage because teams no longer need to train separate, isolated models for images, video, and audio. However, this convenience comes at a severe cost: explainability and control. Because these systems operate under the broader umbrella of large language models (LLMs), practitioners do not have 100% visibility into what is happening internally when generating an output. In high-stakes applications like autonomous driving—where a model might be asked to estimate the probability of an accident from a complex scene—treating the model as a black box poses substantial risks. When an error occurs, diagnosing the exact root cause and identifying specific failure modes remains extremely difficult.
```mermaid
flowchart TD
accTitle: Evolution of ML Model Failure Modes
subgraph S1["Traditional Paradigms"]
direction TB
M1["Custom Large Models<br/>Trained for specific single tasks"]
end
subgraph S2["Generic Multimodal & LLM Paradigm"]
direction TB
M2["Semantic & Multimodal Models<br/>Prompt-driven across audio/vision/text"]
end
Industry failure modes have transformed alongside data annotation practices. Historically, practitioners trained dedicated, custom models for single problems. Today, generic semantic models act on prompts across a wide range of tasks. This shift introduces brand-new challenges:
- **Prompt Sensitivity:** Minor changes to input prompts can drastically alter the quality of generated pre-labels and downstream predictions.
- **Evaluation Metrics:** Measuring data quality now relies heavily on metrics such as PCK (Percentage of Correct Keypoints / Percentage of Correct Points) across varied prompts.
- **Opaque Machine Communication:** As multiple autonomous models communicate and pass data between one another to index systems, they begin developing non-natural-language communication formats that are completely illegible to humans.
- **Robust Evaluation Systems:** To prevent the explainability problem from escalating, teams must implement rigorous evaluation frameworks combining automated validators with human-in-the-loop evals.
---
[31:32](https://www.youtube.com/watch?v=UZv87qrKM54&t=1892s) Managing Compute Budgets, Distillation, and Architecture Trade-offs
With Big Tech hyperscalers spending over $100 billion a quarter on AI infrastructure, managing financial budgets, latency constraints, and GPU allocation is a critical concern for production engineering teams. When deploying applications to hundreds of millions or billions of active users (such as at LinkedIn or Meta), running raw, uncompressed LLMs/VLMs directly in live serving paths is financially and computationally prohibitive.
```mermaid
flowchart LR
accTitle: Production Strategy for High-Scale LLM Serving
subgraph Offline["Offline Stage"]
L1["Large Frontier / Multimodal LLM<br/>High compute & full capabilities"]
end
subgraph Compression["Optimization Layer"]
D1["Knowledge Distillation<br/>Pruning & Quantization"]
end
subgraph Online["Online Production Serving"]
S1["Distilled Compact Model<br/>100% of User Traffic<br/>Low Latency & Cheap GPU Infra"]
end
Offline --> D1
D1 --> Online
```
To balance accuracy against infrastructure spend, teams adopt distinct architectural strategies:
- **Offline LLMs & Online Distillation:** Companies use massive LLMs offline to capture complex reasoning and entity extraction, then use knowledge distillation to train lightweight models for online serving. While raw LLMs might only serve a 1% to 2% experimental canary traffic slice, distilled models allow scaling to 100% of user traffic without inflating GPU infrastructure costs.
- **Binary Search Model Sizing:** In performance-driven environments where engineers do not set a strict monetary budget, teams benchmark target precision (e.g., Average Precision / AP) by experimenting with small and large baseline models. They perform a "binary search" by halving model sizes to observe where performance degrades, identifying the sweet spot between parameter count and inference speed.
- **Upfront Strategy Investment:** Rather than directing all capital to raw compute, organizations should invest early in deliberate data strategy, clear initiative design, and iterative workflows. Underinvesting in research design leads to massive downstream budget waste when unvetted pipelines run in the wrong direction.
- **Specialized Data Funding:** Seed-stage startups in the space (such as Perle, which raised a $17.5M seed round) focus the overwhelming majority of their capital directly on modeling and researching human-machine interaction data.
---
[36:47](https://www.youtube.com/watch?v=UZv87qrKM54&t=2207s) Q&A: Handling Performance Gaps Between Offline and Online Environments
The panel transitions to audience questions with the foundational reminder that "garbage in, garbage out" remains the guiding law of machine learning, making continual stress-testing vital across every stage of a pipeline. The first audience question tackles the discrepancy between offline benchmark evaluations and real-world online production performance.
In offline testing, evaluation datasets are fundamentally static and saturated. Once a model transitions to online production serving, it encounters unseen environmental factors, edge cases, and distribution shifts that degrade performance compared to initial test results. Resolving this delta requires an active, continuous feedback loop: production data must be monitored, gathered, and fed back into the training corpus to keep the training distribution aligned with real-world customer usage patterns.
[38:21](https://www.youtube.com/watch?v=UZv87qrKM54&t=2301s) Measuring Drift and Performance in Rich Computer Vision Data
An audience member opens with a question about quantifying the feedback loop from production back into training. While simple attributes like image resolution are straightforward to check, rich visual data lacks the straightforward statistical distribution metrics readily available for tabular datasets. The questioner asks how teams can systematically detect and quantify distribution shifts and model degradation in vision pipelines.
Vijayraj Gohil explains that while engineering teams routinely track operational metrics like accuracy, precision, and overall performance between offline test sets and live production models, diagnosing the root cause of a performance drop requires deeper inspection. One standard approach involves running A/B testing frameworks alongside automated evaluators that pull production samples to assess similarity against historical baselines. Because computer vision inputs are inherently high-dimensional, teams project both live production data and training datasets into a shared embedding space. Visualizing and measuring distances within these vector representations highlights where live data diverges from the training distribution, providing concrete cues about which parameters or visual characteristics must be collected or synthesized to close the gap.
```mermaid
flowchart TD
accTitle: Production Feedback and Drift Detection Workflow
subgraph DataSources["Data Sources"]
A["Live Production Stream"]
B["Historical Training Data"]
end
subgraph ActionLoop["Mitigation & Retraining"]
E["Identify Divergent Clusters"]
F["Targeted Data Ingestion & Fine-Tuning"]
end
A --> C
B --> C
C --> D
D --> E
E --> F
```
Adding to this approach, another panelist highlights semantic matching directly in the embedding space. Rather than relying purely on raw pixel-level metrics, teams can perform semantic matching between incoming production samples and baseline training data to evaluate feature drift. Quantifying these semantic vector distances enables practitioners to pinpoint the exact distributional changes causing model regressions.
- **Primary baseline monitoring**: Real-time tracking of standard classification and detection metrics (precision, accuracy, latency) across production inference.
- **Embedding space visualization**: Projecting high-dimensional image tensors into latent vector spaces to measure clustering differences and distance anomalies.
- **Semantic drift matching**: Evaluating vector-level feature drift between incoming production imagery and existing training corpora to guide retraining parameters.
---
[40:28](https://www.youtube.com/watch?v=UZv87qrKM54&t=2428s) Training Solely on Synthetic Data and Domain Adaptation Strategies
The discussion transitions to a two-part audience question regarding synthetic data and evolving computer vision paradigms. The attendee asks what concrete strategies practitioners should employ when forced to train models exclusively on synthetic data with virtually no real-world ground truth available. Secondly, they ask about the operational difficulty of migrating from traditional computer vision heuristics to modern deep learning, multimodal architectures, and vision-language models (LLMs).
Vijayraj Gohil notes that relying predominantly on synthetic data is increasingly common rather than an anomaly. When real-world data is scarce or nonexistent, synthetic datasets are best used for domain adaptation on top of strong open-source foundation models. Rather than training a model from scratch purely on synthetic assets, practitioners take a pre-trained backbone that already understands general visual features and use synthetic samples to pivot the model toward the specific target domain. Dimensionality comparisons in embedding space can then reveal discrepancies between the synthetic domain representations and whatever small real-world probe sets exist, allowing human annotators to step in and adjust synthetic parameters.
Teresa O'Neill follows up with pragmatic advice: "Lower your expectations if all you have is synthetic data." She emphasizes that if synthetic data is the sole training material, teams must dramatically constrain their problem statement. Citing an agricultural application akin to automated strawberry picking, O'Neill describes an initiative aimed at identifying specific weed species within soy crop fields. By synthesizing a tightly scoped dataset, the team successfully fine-tuned an existing pre-trained vision model to detect that exact weed in soy canopies. While the model had zero capability outside that exact narrow scenario, it performed reliably within its tight boundary. If teams attempt to apply purely synthetic training across broad, highly dimensional hypothesis spaces, the models inevitably break down.
- **Pre-trained foundation adaptation**: Fine-tuning robust open-source backbones on synthetic samples to shift domain boundaries rather than training weights from scratch.
- **Narrow problem scoping**: Restricting model objectives to specific, constrained tasks (e.g., detecting one weed variety in a single crop type) when ground truth is absent.
- **Embedding alignment**: Comparing real probe samples against synthetic generation batches to identify visual feature disconnects.
---
[43:28](https://www.youtube.com/watch?v=UZv87qrKM54&t=2608s) Ahmed Rashad’s Three Watch-Outs for Synthetic Data and Panel Wrap-Up
Ahmed Rashad warns the audience about recurring pitfalls that lead to suboptimal results when relying on synthetic data pipelines. He outlines three fundamental "watch-outs" that engineering leaders must account for throughout the model lifecycle:
1. **Temporal Degradation and Concept Drift**: Synthetic data pipelines do not remain static. "What's working today very likely will not work tomorrow," Rashad notes, stressing that teams must continuously re-evaluate whether synthetic generation parameters still match live environments over time.
2. **Domain-Specific Outlier and Edge-Case Testing**: It is insufficient to test a model exclusively against the clean data distributions it was trained on. Validation suites must rigorously evaluate performance against real-world exceptions, edge cases, anomalies, and outliers specific to the deployed business use case.
3. **The Human-in-the-Loop Feedback Loop**: In what Rashad characterizes as an ironic reality, generative models that create synthetic datasets consistently perform significantly better when guided and refined by human-labeled inputs.
Rashad concludes by reiterating Teresa's pragmatism: there is no single "magic bullet" in computer vision pipelines. Practitioners must maintain a continuous cycle of empirical testing and adaptation: "If it works, it works, right?... It's not going to be the magic bullet. There's no magic bullet. And even if it's a magic bullet today, tomorrow it won't be. So, just keep experimenting and testing and if it—when it stops working, it stops working. If it works, it works."
The panel concludes with the moderator thanking Ahmed Rashad, Arpita Vats, Saurabh Kumar, Vijayraj Gohil, and Teresa O'Neill for their insights, and acknowledging the audience for their engagement across data labeling, synthetic workflows, and production pipeline management.
## Notable quotes
> [10:05](https://www.youtube.com/watch?v=UZv87qrKM54&t=605s) turning the right knobs at the right time. — Teresa O'Neill
> [22:20](https://www.youtube.com/watch?v=UZv87qrKM54&t=1340s) when the model fails, it fails so hard and so weird. — Teresa O'Neill
> [26:06](https://www.youtube.com/watch?v=UZv87qrKM54&t=1566s) the cost of a failure is significantly higher — Ahmed Rashad
> [36:47](https://www.youtube.com/watch?v=UZv87qrKM54&t=2207s) garbage in, garbage out — Saurabh Kumar
> [40:28](https://www.youtube.com/watch?v=UZv87qrKM54&t=2428s) Lower your expectations if all you have is synthetic data. — Teresa O'Neill
> [43:28](https://www.youtube.com/watch?v=UZv87qrKM54&t=2608s) What's working today very likely will not work tomorrow — Ahmed Rashad
> [43:28](https://www.youtube.com/watch?v=UZv87qrKM54&t=2608s) If it works, it works, right?... It's not going to be the magic bullet. There's no magic bullet. And even if it's a magic bullet today, tomorrow it won't be. So, just keep experimenting and testing and if it—when it stops working, it stops working. If it works, it works. — Ahmed Rashad
## Mentioned
Arpita Vats, Saurabh Kumar, Vijayraj Gohil, Teresa O'Neill, Ahmed Rashad, LinkedIn, Meta, Uber, Kraft Foods, 2K Games, New York University (NYU), iMerit, Perle, Scale AI, Amazon, McKinsey, MIT, Amazon Go, CVPR, HIPAA
## Video description
Arpita Vats, Senior AI Engineer, LinkedIn
Saurabh Kumar, Senior Manager, Ex-Meta, Uber, Kraft Foods
Vijayraj Gohil, Machine Learning Engineer, 2K Games
Teresa O'Neill, Director of Solutions Architecture, Natural Language Understanding, iMerit Inc.
Ahmed Rashad, Founder & CEO, Perle
Abstracts
This session will focus on the importance of building robust data pipelines for computer vision projects, with an emphasis on data labeling, management, and strategy. Panelists will discuss the challenges of curating large-scale datasets, ensuring high-quality labeled data, and implementing efficient data workflows to support model training and optimization. Learn how to structure and maintain a strong data pipeline that supports continuous learning and improvement in computer vision models. The discussion will also cover data augmentation, synthetic data, and best practices for handling edge cases and imbalanced datasets.
Robust computer vision pipelines depend less on generic automation or synthetic silver bullets and more on strict data lineage, modular verification against physical ground truth, and empirical hypothesis testing. Panelists argue that while multimodal architectures expand reasoning capabilities, sustaining production reliability requires continuous embedding-level drift monitoring, architectural distillation, and disciplined human-in-the-loop governance.
Ground Truth Rigor and Failure Attribution (1:48)
Arpita Vats and Teresa O'Neill distinguish computer vision from natural language processing by noting that visual pipelines capture measurable physical realities governed by knowable coordinates and physical laws. This physical grounding permits objective, mathematically rigorous quality control standards that subjective, context-dependent text workflows cannot support. However, high-dimensional visual inputs remain fragile: while deep neural networks can absorb baseline architectural noise, corrupted pixel inputs and uncurated out-of-distribution imagery compromise downstream feature extraction.
Ahmed Rashad emphasizes that pipeline failures frequently trigger attribution confusion across operations teams. When model degradation occurs, practitioners often prematurely blame human labeling variance rather than systematically investigating the entire lifecycle. In many cases, model degradation stems from upstream data collection biases, inadequate sample volume, or out-of-sequence ingestion rather than annotator error.
Data Contracts and Modular Lineage Verification (6:26)
To prevent data inconsistencies from silently destabilizing models, Vijayraj Gohil implements strict data contracts alongside drift monitors. In production gaming environments, he enforces a lineage-forced promotion gate that automatically blocks any model deployment if a detected performance shift cannot be traced directly back to a validated data slice and its historical lineage.
The panel cautions against waiting for end-to-end model evaluation before auditing data, advocating instead for modular verification at every intermediate pipeline stage. Teresa O'Neill explains that partnering closely with annotation operations serves as a crucial early warning system. By indexing raw data in batches, human annotators can surface severe real-world distribution anomalies—such as erratic flood driving or cultural festival traffic patterns in autonomous vehicle data—before organizations commit significant GPU compute to training runs.
The Deliberate Scoping of Synthetic Data (9:11)
Teresa O'Neill and Ahmed Rashad warn against treating synthetic data generation as a panacea, noting that synthetic pipelines reproduce predictable patterns but consistently fail to capture genuine real-world edge cases and uncodified expert knowledge. While synthetic data successfully mitigates privacy restrictions in regulated settings—such as simulating multilingual clinician-patient dialogues—and can address narrow, tightly bounded tasks like identifying specific weed varieties within soy crops, attempting to apply it across broad hypothesis spaces causes models to break down.
When real-world ground truth is unavailable, Vijayraj Gohil advises using synthetic data primarily for domain adaptation on top of pre-trained foundation backbones rather than training weights from scratch. Rashad outlines three critical operational checks for synthetic workflows: continuously tracking temporal parameter drift, stress-testing models against domain-specific edge cases rather than clean training distributions, and maintaining active human-in-the-loop feedback to calibrate generative outputs.
Multimodal Serving, Distillation, and Latent Drift (31:32)
As computer vision shifts toward vision-language models to generate explanatory 'talk tracks' for autonomous decision-making, the transition introduces severe black-box interpretability challenges and prompt sensitivity. To manage high inference latency and infrastructure spend when serving large user bases, engineering teams adopt an asymmetric architecture: large foundation models operate offline for complex reasoning and extraction, while knowledge distillation trains lightweight, compact models to serve 100% of live online traffic.
Because visual data lacks the straightforward scalar distribution metrics of tabular data, teams cannot rely on basic pixel statistics to diagnose production degradation. Gohil details projecting live inference streams and baseline training corpora into shared high-dimensional embedding spaces. Measuring vector distances and semantic feature drift within these latent spaces allows practitioners to isolate divergent clusters and systematically source the exact imagery required for retraining.
The panel emphasizes that engineering computer vision systems requires treating data operations as a hypothesis-driven empirical discipline rather than relying on rigid playbooks or uncalibrated automation. As multimodal architectures increase pipeline opacity, the central unresolved challenge is balancing compute budgets and inference speed against the demand for interpretable decision-making and continuous, human-guided alignment with physical reality.
## Background
该内容由频道“Ai4”于 2025 年 8 月 27 日发布在 YouTube 平台。视频记录了一场关于构建高可用计算机视觉（CV）数据管道的圆桌研讨会，来自 LinkedIn、2K Games、iMerit 和 Perle 等企业的资深 AI 工程师与管理者围绕数据标注策略、数据集管理、合成数据应用及边缘案例处理等核心议题展开了深入探讨。
## TL;DR
研讨会首先剖析了计算机视觉管道（建模物理现实）与大语言模型管道（建模认知现实）的根本差异，并深入探讨了从数据采集、版本控制到精准根因归因的工程挑战。专家们详细阐述了在黄金测试集与大规模量产阶段如何平衡人工标注与模型预标注，并提出了数据契约、血缘晋级机制及多阶段解耦验证等防护方案。针对合成数据与多模态趋势，嘉宾们指出合成数据无法替代真实的未知长尾异常，必须审慎控制假设空间，同时建议通过语言推理为自动驾驶等黑盒模型赋予可解释性。最后，讨论给出了利用嵌入空间与语义匹配量化线上数据漂移的具体手段，并分享了在严苛算力约束下通过知识蒸馏与闭环反馈实现模型持续迭代的实战经验。
## Atmosphere & style
整场研讨会散发着务实、严谨且极具工业界落地色彩的工程师思辨氛围，嘉宾们毫不避讳技术痛点，直言“不要迷信所谓的行业最佳实践”。对话节奏紧凑且富有层次，既有底层数据哲学的深入辨析，又穿插着自动驾驶街头遭遇孔雀、零售手部遮挡图像修复、大豆田杂草识别等生动鲜活的实战案例。研讨完全剥离了浮躁的 AI 营销话术，在理性的技术交锋与经验共享中，展现出顶尖开发者在面对模型黑盒与物理现实鸿沟时的清醒与克制。
## Key takeaways
- [01:48](https://www.youtube.com/watch?v=UZv87qrKM54&t=108s) 计算机视觉（CV）管道旨在对客观的物理现实建模，具有明确的 Ground Truth 与严格量化的质检标准；而大语言模型（LLM）则建模认知与社会现实，评估更容易主观化。
- [04:02](https://www.youtube.com/watch?v=UZv87qrKM54&t=242s) CV 数据管道排障的核心痛点并非发现表面故障，而是精准归因根因，避免将样本数量不足或摄取时机偏差误判为标注质量问题。
- [06:26](https://www.youtube.com/watch?v=UZv87qrKM54&t=386s) 模型预标注与人工标注需按阶段动态权衡：构建黄金测试集必须高度依赖人工以保客观真值，而大规模预训练则重度依靠自动预标注降低成本。
- [10:05](https://www.youtube.com/watch?v=UZv87qrKM54&t=605s) 合成数据并非万能药，擅长解决冷启动和隐私敏感场景（如医疗医患对话），但具有先验局限性，无法模拟人类认知边界之外的真正长尾未知边缘案例。
- [14:52](https://www.youtube.com/watch?v=UZv87qrKM54&t=892s) 成熟的 CV 数据管道应采用分阶段模块化验证，并在标注第一线建立早期预警系统，在数据进入高成本 GPU 算力消耗前即时拦截环境噪音与分布异常。
- [19:26](https://www.youtube.com/watch?v=UZv87qrKM54&t=1166s) 工业界与学术界在 CV 模型上的诉求不同：工业界聚焦基础大模型的微调适配、组合融合与高吞吐工程架构，学术界则侧重原生能力上限与底层数据贡献归因。
- [22:20](https://www.youtube.com/watch?v=UZv87qrKM54&t=1340s) 将视觉感知系统与语言推理模型融合为驾驶意图提供“实时解说”，能有效打破自动驾驶黑盒状态，帮助工程师定位离奇决策与长尾边缘用例。
- [28:46](https://www.youtube.com/watch?v=UZv87qrKM54&t=1726s) 多模态与 LLM 体系的提示词演进带来了黑盒可解释性危机，甚至演化出非自然语言通信协议，必须结合人工反馈与自动验证器构建混合评估矩阵。
- [31:32](https://www.youtube.com/watch?v=UZv87qrKM54&t=1892s) 在企业算力与预算约束下，工业界通常采用离线大模型加知识蒸馏提炼至在线小模型，或通过二分搜索折半测试快速锁定性价比最优的模型尺寸。
- [36:47](https://www.youtube.com/watch?v=UZv87qrKM54&t=2207s) 解决线上服务与离线评估性能断层的关键是建立端到端反馈闭环，持续捕获线上低置信度异常样本并反哺重训模型。
- [38:21](https://www.youtube.com/watch?v=UZv87qrKM54&t=2301s) 量化衡量高维计算机视觉数据漂移的核心方法是将生产在线数据与训练集投影至嵌入空间，通过特征向量可视化与语义匹配度量分布偏移。
- [40:28](https://www.youtube.com/watch?v=UZv87qrKM54&t=2428s) 若仅依赖合成数据训练，最佳实践是利用开源基座模型微调迁移并极度收窄任务假设空间，同时需保持持续压力测试并引入人工数据反馈形成迭代闭环。
## Detailed walkthrough
### [00:04](https://www.youtube.com/watch?v=UZv87qrKM54&t=4s) 嘉宾背景介绍
研讨会主持人 Saurabh Kumar 宣布讨论正式开始，并邀请四位嘉宾逐一进行自我介绍，为后续关于计算机视觉数据管道的深入探讨拉开序幕：
- **Vijayraj Gohil**：现任 2K Games 机器学习工程师，此前曾在零售科技（retail tech）及机器学习领域工作，并在纽约大学（NYU）担任过机器学习研究员。
- **Teresa O'Neill**：语言学家、数据运营专家，现任 iMerit 生成式 AI（GenAI）业务总监。iMerit 专注于为汽车、医疗、企业级生成式 AI 等行业提供专业数据服务及 AI 数据转换。
- **Arpita Vats**：领英（LinkedIn）资深 AI 工程师。她在攻读博士学位期间主攻计算机视觉（CV）与自然语言处理（NLP），在过去三到四年中专注于大语言模型（LLM）与生成式 AI（GenAI）领域的研究与工程落地。
- **Ahmed Rashad**：Perle AI 创始人兼 CEO。在创立公司之前，他曾于 2018 年至数年前在 Scale AI 建立并领导人类数据运营业务（Human Data Operations），更早之前在亚马逊（Amazon）和麦肯锡（McKinsey）从事 AI 应用开发与咨询。他曾在麻省理工学院（MIT）长期攻读运筹学，值得一提的是，其职业生涯最早始于海上钻井平台（offshore driller），随后才转型进入人工智能领域。
---
### [01:48](https://www.youtube.com/watch?v=UZv87qrKM54&t=108s) 计算机视觉与传统/LLM 数据管道的核心差异
主持人抛出了第一个议题：计算机视觉（CV）数据管道与大多数人熟悉的传统管道（如 NLP 管道）有何本质不同？
Arpita Vats 指出，相较于通常只处理纯文本或语音转文本的 NLP 管道，计算机视觉管道要复杂得多。CV 管道需要承载更丰富的多模态信息，不仅包含图像与视频流，还要同步整合音频、时间戳以及字幕等全方位元数据。因此，处理视觉任务时，数据管道必须具备极高的多维度承载力与更为严密的架构设计。
Teresa O'Neill 进一步从底层认知与哲学层面进行了补充。她强调，计算机视觉管道与 LLM 管道在根本目标上存在哲学差异：
- **计算机视觉（CV）**：旨在为“物理真实世界”建模。真实世界是可感知的、存在绝对客观的 Ground Truth（如物体的存在、确切空间坐标、运动轨迹与行为模式）。这使得 CV 能够在质量控制（QC）、精度定义和需求设计阶段实施极其客观且严格的标准。
- **大语言模型（LLM）**：旨在建模“认知现实”或“社会现实”。这些领域往往缺乏唯一的客观真值，导致质量把控极易滑向主观化，边界难以精确量化。
```mermaid
flowchart TD
accTitle: CV与LLM数据管道建模差异对比
subgraph CV["计算机视觉 (CV) 数据管道"]
A1["现实物理世界建模"] --> B1["具有明确客观 Ground Truth<br/>（位置、运动、像素、类别）"]
B1 --> C1["可执行高度严格、客观的质检与需求定义"]
end
subgraph LLM["大语言模型 (LLM) 数据管道"]
A2["认知与社会现实建模"] --> B2["缺乏绝对单一客观真值<br/>（语言表达、主观偏好）"]
B2 --> C2["质检与评估容易主观化，标准难以量化"]
end
```
---
### [04:02](https://www.youtube.com/watch?v=UZv87qrKM54&t=242s) 视觉管道最易出错的环节与根本原因排查
针对主持人关于“CV 数据管道中最容易发生故障的环节（数据采集 Ingestion、数据标注 Labeling、版本控制 Versioning 还是监控 Monitoring）以及相应防护措施（Guardrails）”的提问，嘉宾们分享了各自的实战洞察。
Arpita Vats 强调了“数据质量”与“版本控制”的双重挑战：
- **数据与像素质量**：模型对数据中的噪声极其敏感。如果模型结构本身存在轻微噪声尚可应对，但如果输入的图像数据或像素质量存在噪声或分布偏差，模型将彻底无法解析。
- **数据版本控制**：线上真实世界的数据分布随时在发生变化。如果缺乏严密的版本控制，用于更新模型的训练集分布就会迅速落后于生产环境实际见到的数据，从而导致模型线上表现恶化。
Ahmed Rashad 提出了更深层的工程痛点：在完整的数据链条中，最难的往往不是“故障在哪个环节爆发”，而是**“精准定位引发故障的根本原因（Root Cause）”**。在实际工程中，团队经常混淆表象与根本问题——例如，模型效果变差可能被误判为“标注质量低下”，但根因其实是“最初采集的数据样本根本不对”、“样本数量严重不足”或者“数据摄入时机存在偏差”。虽然人类标注员的高变异性（Variability）会引入更多潜在失效点，但如何准确归因才是整个系统中最具挑战性的环节。
---
### [06:26](https://www.youtube.com/watch?v=UZv87qrKM54&t=386s) 调节管道控制参数：在模型预标注与人工标注之间权衡
在探讨项目权衡指标（交付周转 SLA、质检协议、成本控制）以及“模型预标注（Pre-labeling）vs 纯人工标注（Human Labeling）”的相对表现时，Ahmed Rashad 警示大家不要陷入非黑即白的误区。许多客户会迷信某种单一技术（如合成数据），但没有一种方案是放之四海而皆准的灵丹妙药。
Ahmed 强调，最佳策略完全取决于当前所处的研发阶段：
- **构建黄金测试集（Golden Dataset）时**：必须极度重度依赖人工标注，仅辅助极少量的机器介入，以保证绝对的高精度与客观真值。
- **大规模预训练（Pre-training）或海量量产时**：则需要重度依赖模型自动标注/预标注，将人工介入降到最低以控制成本并提高吞吐效率。
Vijayraj Gohil 随后从数据架构落地层面进行了补充，指出行业长期面临**“数据标注偏差”**与**“数据架构演进（Schema Evolution）”**两大顽疾。他给出的工程实践方案包括：
- **制定数据契约（Data Contracts）**：规范数据输入与输出规范，防止格式与定义失控。
- **部署漂移监控器（Drift Monitors）**：在每次模型使用新数据训练时，持续监测性能与特征漂移。
- **强制血缘晋级机制（Lineage Force Promotion）**：建立严密的数据与模型血缘关系追踪。如果无法追溯某次性能表现是由哪些具体数据驱动的，该模型就绝对不允许发布到生产环境。
---
### [09:11](https://www.youtube.com/watch?v=UZv87qrKM54&t=551s) 合成数据与数据增强的现实鸿沟（Reality Gap）
主持人提问：是否有某个具体任务中，合成数据（Synthetic Data）或数据增强（Augmentation）真正补齐了长尾边缘案例（Long-tail Edge Cases）的鸿沟，又是在哪些场景中彻底失败？团队该如何验证这种“现实鸿沟”，避免对生成器（Generator）产生过拟合？
Teresa O'Neill 明确表示，尽管业界对合成数据抱有极大热情，但它绝不是包治百病的灵丹妙药（Panacea）。使用合成数据必须建立经过审慎考量的设计闭环：团队必须想清楚“为什么要合成这批数据”、“试图弥补哪类具体缺失”，以及“当前工具是否真正适合填补这一鸿沟”。
她以医疗健康（Healthcare）场景为例指出，合成数据在某些特定边缘案例下能够发挥巨大价值，但整个流程中依然离不开严格的人类专家监督与闭环介入。
### [10:05](https://www.youtube.com/watch?v=UZv87qrKM54&t=605s) 合成数据的适用边界与真实长尾困境
Teresa O'Neill 结合医疗大语言模型（LLM）的具体案例，深入阐述了合成数据（Synthetic Data）的合理定位。在医疗场景中，由于数据可用性匮乏以及极其严苛的隐私限制，团队无法轻易获取真实世界的临床数据，因此他们合成了横跨多个医学亚专科、覆盖多种语言的医患对话交互数据集。虽然该方案有效规避了 HIPAA 隐私合规风险、填补了真实数据的空白，且模型表现良好，但整个过程依然需要大量的人工严密监控。Teresa 强调，合成数据的引入必须非常谨慎且深思熟虑，需将其放置在模型开发生命周期的正确节点，在准确性、安全性和成本之间权衡取舍——正如 Ahmed 所言，这本质上是一门“在正确的时间调节正确的旋钮”的平衡艺术。
```mermaid
flowchart TD
accTitle: 合成数据与真实数据在管道中的权衡与定位
subgraph SyntheticPhase["合成数据阶段 (适用：冷启动 / 隐私敏感)"]
A["合成交互生成<br/>(模拟多语种/多专科对话)"] --> B["规避 HIPAA / 隐私风险"]
B --> C["冷启动 PoC 验证与原型开发"]
end
subgraph RealWorldPhase["真实世界校验阶段 (必需：专家与人工闭环)"]
C -.-> D{"面对未知长尾/边缘用例?"}
D -- "合成数据存在先验局限" --> E["引入领域专家 (医生/法务)"]
E --> F["人工持续监控与微调"]
F --> G["沉淀真实长尾知识库"]
end
```
Ahmed Rashad 对此表示强烈赞同，并指出“万事万物皆有其时其地，绝大多数错误都源于在错误的时间将工具放错了位置”。合成数据在搭建概念验证（PoC）时固然出色，但其本质是基于既有模型生成的“可预测数据”，无法百分之百复现复杂现实。在医学、法律等人类全部知识尚未被 AI 完全摄取的领域，合成数据甚至会凭空制造出大量脱离实际的虚假边缘用例（Edge Cases），反而漏掉真正的现实异常。例如，面对一个连多位资深医生都无法给出明确诊断结论的疑难病例，合成数据受限于既有规则与已知知识，永远无法自主生成此类真实的未知场景。
- 合成数据核心价值：解决真实数据在冷启动时期的样本空白，有效规避 HIPAA 等强监管领域的隐私合规隐患。
- 局限性与风险：合成样本往往具有先验确定性，无法模拟人类认知边界之外的真正长尾场景与争议性边缘用例。
---
### [12:27](https://www.youtube.com/watch?v=UZv87qrKM54&t=747s) 数据版本控制、血缘追踪与科学迭代法
主持人 Arpita Vats 将话题引向计算机视觉数据管道的核心工程架构，询问嘉宾们在数据集、标注标签和模型三者之间如何构建版本控制与数据血缘关系（Data Lineage Contract）。Arpita 特别指出，当标注规范（Label Schema）出现 Bug 或管道故障时，团队需要多快能够复原“哪个模型在何时看到了哪些数据”，以便从源头精准定位并修复问题。
Ahmed Rashad 率先回应，他直言开发者不应盲目为了所谓“最佳实践”而进行大规模无差别清洗或标注，“谁在乎最佳实践？如果你正在构建前所未有的创新系统，最佳实践往往根本不适用”。他提倡采用严谨的“科学方法”来对待数据管道：对每次改动设定明确的预期输出值，一旦实际结果偏离预期，必须立即暂停并回溯根因。问题通常出在机器、人工或流程这三个维度之一。Ahmed 强调，高频的快速迭代远比单纯堆砌数据量更重要，顶尖工程师与普通开发者的核心差距就在于能否做到“每日微调”甚至单日内完成数百上千次细微调整，而不是以周为周期拖延排查。
---
### [14:52](https://www.youtube.com/watch?v=UZv87qrKM54&t=892s) 管道分阶段验证与标注运营的早期预警系统
Vijayraj Gohil 对 Ahmed 的观点进行了补充与辨析。他认为，虽然快速回溯非常关键，但无需等整条长管道全量跑完后才去追查失败原因。成熟的 CV 管道应当划分为多个解耦的独立阶段（Phases），每个阶段都应具备确定的预期输出指标（如特征分布或中间产物形态）。通过对各中间阶段进行逐步验证，工程师可以在局部即时阻断异常，避免耗费一两周跑完整体流程后陷入漫长而痛苦的全局回溯。Saurabh Kumar 随后补充道，在排查数据污染与 Bug 时，既需要验证自动化工具的构建逻辑与生成机制，也需要针对人工标注人员设计验证题集，摸清标注员在作业前的初始假设，从而系统化锁定错误来源。
```mermaid
flowchart LR
accTitle: CV 数据管道多阶段验证与标注运营早期预警机制
subgraph Ingestion["数据摄取与分阶段验证"]
A["原始数据批次"] --> B["阶段输出分布校验"]
end
subgraph EarlyWarning["标注运营 (Early Warning System)"]
B --> C{"人工/自动化质量审计"}
C -- "发现真实世界异常<br/>(如极端天气/节日驾驶)" --> D["即时阻断 & 修正 Schema"]
C -- "质检通过" --> E["批次索引与血缘追踪"]
end
subgraph Training["模型训练阶段"]
E --> F["预训练 / 微调 / RLHF"]
D -. "避免无效算力浪费".-> F
end
```
Teresa O'Neill 则从“标注运营管理”（Labeling Operations Management）的角度，分享了如何借鉴传统制造业与服务业的管理规范来构建“早期预警系统”（Early Warning System）。当数据在缺乏充分清洗就直接打包交付时，专业标注团队往往能最先察觉现实世界的异常模式。例如，在自动驾驶视觉数据中，标注员会迅速发现“某些批次是在特定节日期间采集的，当地驾驶行为完全反常”，或者“车辆遭遇了突发特大暴雨与道路积水，而现有算法并未对此建模”。借助严密的数据批次索引、血缘追踪机制以及独立的质量审计流程，团队可以在人工标注环节第一时间捕获异常并纠偏，从而避免在后续阶段浪费昂贵的 GPU 算力，或等到模型部署上线发生漂移（Drift）后才仓促应对。
- 管道模块化校验：在数据管道各阶段设立明确的期望分布校验点，将排障范围控制在局部单元内。
- 标注运维作为哨兵：利用人工标注第一线的敏锐洞察，在模型训练前精准拦截诸如节假日驾驶异常、极端天气涉水等环境长尾噪音。
- 审计与成本控制：在数据进入高成本算力消耗阶段前，依靠独立的数据血缘与质量审计完成纠错，大幅削减试错成本。
---
### [18:56](https://www.youtube.com/watch?v=UZv87qrKM54&t=1136s) 实际计算机视觉管道的应用场景探讨
在厘清了数据管道的设计哲学与质量管控之后，主持人 Arpita Vats 转向具体业务落地，向全体嘉宾抛出了实践层面的核心问题：大家在过往的工业界实战中，究竟是出于何种业务目标而构建计算机视觉数据管道的？各自经历中最常见、最具代表性的 CV 应用场景又是什么？嘉宾们随即结合自身在自动驾驶、游戏渲染及互联网大规模推荐与内容理解等领域的工程经历展开了逐一拆解。
### [19:26](https://www.youtube.com/watch?v=UZv87qrKM54&t=1166s) 零售结账中的图像修复与工业界与学术界的研究差异
Vijayraj Gohil 分享了他在学术研究时期参与的自动化零售结账系统项目。该系统类似于 Amazon Go，用户在店内自行挑选商品，无需人工收银员进行结算，但在用户把商品放在自助结账台时会面临严重的计算机视觉挑战。结账台上堆放了大量杂物，周围存在各种噪声干扰，最主要的是用户在拿取或扫描商品时手部会遮挡物体。针对这个问题，团队开发了一种图像修复（Inpainting）算法，用于消除手部遮挡和背景噪声，还原被遮蔽的区域。
由于训练集大多为 2D 图像，而现实中的商品是 3D 实体，因此模型必须能够从不同角度和对比度特征中准确识别出特定的物体 ID。系统最终成功解决了多角度识别与去噪问题，这主要得益于训练数据覆盖了同一物体的各种视角，即使存在严重的手部重叠或遮挡，模型依然能够稳定完成识别。
Vijayraj 进一步对比了工业界与科研界在面对计算机视觉基础模型时的不同侧重点：
```mermaid
flowchart TD
accTitle: 工业界与研究界在计算机视觉任务中的核心差异
subgraph Industry["工业界落地 (Industry Focus)"]
A["基础大模型 (Foundation Models)"] --> B["微调与适配 (Fine-tuning & Retraining)"]
B --> C["组合模型架构 (分割 + 检测多模型融合)"]
C --> D["工程与基础设施 (Infra / 在线采样 / 性能监控)"]
end
subgraph Research["科研探索 (Research Focus)"]
E["基础模型底层能力 (Raw Capabilities)"] --> F["提升分割与检测精度上限"]
F --> G["数据归因研究 (分析哪些数据贡献了特定能力)"]
end
```
- **工业界的核心诉求**：聚焦于工程落地与基础设施架构。工程团队通常直接调用现有的基础模型，针对具体业务领域进行微调（Fine-tuning）和再训练，将图像分割（Segmentation）和目标检测（Object Detection）等多个模型结合使用来解决业务问题。关键挑战在于海量数据处理吞吐量、在线采样机制（Online Sampling）以及测试与生产环境中的模型性能监控。
- **学术界的核心诉求**：聚焦于提升基础模型本身的底层原生能力（Raw Power），如突破通用分割和检测的精度上限，并深入探索到底是哪些具体的数据样本在模型特定能力的形成中起到了决定性贡献。
---
### [22:20](https://www.youtube.com/watch?v=UZv87qrKM54&t=1340s) 多模态与可解释性：用语言模型为自动驾驶黑盒“解说”
Teresa O'Neill 指出，iMerit 在自动驾驶与移动出行领域的长期实践中，观察到了一个明显的演进方向：从单一感知转向多模态融合，并且极度重视系统的信任度与可解释性（Trust and Explainability）。现有的目标检测、轨迹预测和行为模型已经相当成熟，但一旦发生故障，其失败模式往往“极度离奇且严重”（"fails so hard and so weird"），让人完全无法理解模型为何会做出异常决策。
为了提升安全性和可信度，部分客户开始将视觉感知系统与语言推理模型结合，让模型能够以人类语言实时“解说”自己的驾驶意图与行为决策链。例如，模型可以输出内部独白：“我正在沿街行驶，观察到前方有一名行人似乎有乱穿马路（Jaywalk）的意图，因此我决定减速。”这种实时的思维轨迹（Talk Track）直接打破了黑盒状态，让工程师能够清晰排查模型到底在哪个环节出现了认知偏差。
Teresa 举了一个生动的案例：如果系统误将橱窗中的行人倒影当成了真实行人，或者在迈阿密街头遇到了一只孔雀——在迈阿密，孔雀会直接大摇大摆走到路中央停下，摆出一副“我现在拥有这条路”（"I own the road now"）的姿态，完全不像普通鸟类或宠物狗那样遇车避让。如果模型将其误判为牵绳的狗或其他鸟类，就会引发错误的应对策略。通过语言生成的解释，团队能够迅速定位这种长尾边缘案例（Edge Case），有针对性地补充强化学习（RL）或设计专门的测试场景来进行风险兜底。
---
### [24:32](https://www.youtube.com/watch?v=UZv87qrKM54&t=1472s) 具身智能与农业采摘：多模态推理的落地场景
Ahmed Rashad 探讨了计算机视觉向更复杂场景与具身智能（Robotics）演进的趋势。他认为自动驾驶的基础感知问题在很大程度上已经被攻关得差不多了（"more or less cracked"），未来的爆发点将集中在需要高度上下文判断的工业与家用机器人上。
以农业中的草莓采摘机器人为例，系统不仅要识别草莓的位置，还要根据色泽与外观深度判断草莓的成熟度（Ripeness），进而做出采摘、装箱的精细操作决策。在此过程中，机器人需要接受连续的反馈闭环（例如“采摘的这颗草莓其实并未成熟，原因在于……”）。这种复杂的认知判断与反馈学习，单纯依靠传统视觉模型难以完成，必须深度融合语言模型提供的上下文推理与评估能力。
---
### [26:06](https://www.youtube.com/watch?v=UZv87qrKM54&t=1566s) 早期多模态管线的主要失败模式与人工标注的一致性困境
针对主持人关于多模态系统在生产环境中常见失败模式的提问，Ahmed Rashad 将核心问题归结为数据基础层面的两大致命缺陷：
1. **数据不匹配**：从一开始就没有采集到正确的业务场景数据。
2. **正确数据量匮乏且盲目扩增**：在缺乏高质量真实数据时，团队急于使用合成数据（Synthetic Data）或其他来源的数据进行增强，结果引入了更多偏见和噪声。Ahmed 强调，在模型开发的早期阶段，失败代价呈对数级（Log Scale）放大，模型尚未具备足够的泛化鲁棒性，早期的每一次数据错误都会带来十倍以上的负面影响。
对于如何将数据转化为“正确数据”，数据标注、清洗、评测与人工介入不可或缺，但大规模人工参与本身也是最脆弱的环节：
- **人类表现的不稳定性**：大规模人工标注团队很难在长时间内保持绝对的标注一致性。普通人每天起床的状态和情绪都会有所波动，平时一两分钟的偏差对日常生活无伤大雅，但在对数据极度敏感的模型冷启动阶段，这种微小的标注标准漂移就会严重污染数据集。
- **系统与人员建设的关键性**：在项目早期，将合适的人员配置在正确的质检与标注流程体系中，其重要性丝毫不亚于数据采集本身。
Vijayraj Gohil 补充道，从 CVPR 等顶级计算机视觉会议的研究趋势来看，当前学术界几乎全面转向探索大语言模型（LLM）的应用，尤其是在场景描述与自动驾驶高阶规划领域。然而，行业目前仍面临一个根本性的困惑：LLM 内部的真实运作机制依然相当模糊，理解并解释大模型内部的具体表征依然是一个亟待解决的课题。
### [28:46](https://www.youtube.com/watch?v=UZv87qrKM54&t=1726s) 多模态与大语言模型的黑盒挑战：提示词演进与可解释性危机
多模态大模型的快速普及极大地简化了计算机视觉与数据标注的工作流。以往工程师需要为图像、音频、视频等不同模态分别训练独立的专用模型，而如今统一的多模态模型能够一站式处理各种任务。然而，这种便利性也带来了显著的技术代价。多模态模型本质上仍然归属于大语言模型（LLM）的体系框架，其运作机制在很大程度上依然是一个“黑盒”。例如在自动驾驶场景中，向模型输入一个复杂场景并询问“这个人发生交通事故的概率有多大”，系统无法完全掌控模型内部确切的推理过程，排查错误原因（Debug）和定位系统痛点变得异常困难。
从行业失败模式（Failure Modes）的发展历程来看，数据标注模式的演进带来了全新的工程挑战。行业最初依靠针对单一问题定制训练的大型模型，逐步过渡到如今基于通用语义与提示词（Prompt）的大语言模型。但这种转变引入了新的变量：提示词的设计直接左右了预标注（Pre-labeling）的质量。一旦提示词发生微小调整，模型输出的正确关键点百分比（PCK, Percentage of Correct Keypoints）就会随之波动。如何针对多样化的提示词建立有效的验证器（Validators）与评估系统，已成为当下亟待攻克的核心难题。
```mermaid
flowchart TD
accTitle: 从专用模型到多模态提示词驱动的工作流演进
subgraph Traditional["传统架构"]
A["图像专用模型"]
B["音频专用模型"]
C["视频专用模型"]
end
subgraph Multimodal["现代多模态大模型体系"]
D["统一多模态 LLM 底座"]
E["Prompt 提示词工程"]
F["预标注输出 Pre-labels"]
G["黑盒可解释性问题与 PCK 评估"]
end
Traditional -->|整合与演进| D
E --> D
D --> F
F --> G
```
### [30:43](https://www.youtube.com/watch?v=UZv87qrKM54&t=1843s) 构建强健评估体系：应对模型间非自然语言通信的潜在风险
针对模型的黑盒特征与多模态复杂性，构建高质量的评估系统（Evaluators）成为维系模型可解释性与安全性的重中之重。如果不建立强有力的评测机制，可解释性问题将“在好转之前急剧恶化”。当前工业界已经出现多个模型相互调用、相互索引系统并交互传递数据的场景，模型之间甚至演化出了高度精简且高效、但人类完全无法解读的非自然语言通信协议。这种机器自生协议导致系统内部愈发不透明，潜在风险正在逐步累积。
为了确保复杂多模态系统的安全可控，团队必须引入全方位的评测矩阵：
- **混合评估体系**：深度融合人类反馈评估（Human Evals）与自动化验证器（Auto Validators），对跨模态输出进行交叉检验。
- **新兴验证工具集成**：采用新涌现的检测与分析工具，实时监控多模型交互过程中的数据流动。
- **可解释性兜底**：在模型完全黑盒化之前，通过严格的评测基准锁定异常波动，防范模型间通信异化带来的系统性风险。
### [31:32](https://www.youtube.com/watch?v=UZv87qrKM54&t=1892s) 算力预算约束下的模型优化：知识蒸馏与二分搜索选型策略
在科技巨头单季度在人工智能基础设施上投入超过千亿美元的宏观背景下，各企业在落地具体业务管线时面临着严峻的财务与算力约束。对于拥有数十亿活跃用户的大型平台（如 LinkedIn、Facebook），直接在全量线上服务中实时运行高参数量 LLM 会导致 GPU 算力成本失控。目前工业界的常规做法是仅对 1% 到 2% 的线上流量尝试大模型直连，而面向 100% 全量用户时，核心策略是将大模型放在离线环境（Offline）运行，再利用**知识蒸馏（Knowledge Distillation）**技术将其核心特征提炼至小型在线模型中。这样既能在极低推理延迟和有限 GPU 资源下完成在线服务，又能完整继承大模型卓越的高阶表现。
在游戏研发与特定视觉任务场景中（如 2K Games），团队面对的具体目标往往是追求极限精度（AP）与低延迟的平衡。工程师通常不直接设定固定的财务预算上限，而是先制定明确的精度目标，再通过极端实验进行“二分搜索”式选型：
1. **基准建立**：分别在极小模型与超大模型上进行实验，标定性能上下界。
2. **二分搜索折半测试**：将模型尺寸减半，量化评估精度与召回率的下降幅度。
3. **阈值锁定**：如果设定的性能阈值在 1 到 10 之间，重点探索 5 到 7 区间内的模型架构，以最快速度找到性价比最优的模型规模。
对于数据基础设施服务商（如 iMerit），核心观点则是“不要在设计与战略规划阶段过度节省预算”。相比于动辄数百万美元的 GPU 算力开销，前期的研发意图梳理、目标设定与数据策略设计成本占比极小。如果前期在方案设计上投入不足，后期一旦发现研究方向或业务需求偏差，将引发灾难性的算力浪费；建立敏捷迭代的数据设计机制，是防止算力支出失控的最有效防线。而作为数据初创企业（如 Perle），其完成的 1750 万美元种子轮融资则绝大部分直接投入于人机交互数据探索与模型研发中。
### [36:47](https://www.youtube.com/watch?v=UZv87qrKM54&t=2207s) 线上与离线性能断层应对：闭环数据反馈与模型重训
在进入观众问答环节后，主持人总结了核心共识——“垃圾进，垃圾出（Garbage in, garbage out）”在 AI 时代依然成立，对模型各阶段的输出进行极限压力测试至关重要。随后，针对现场观众提出的“如何评估并解决模型离线测试（Offline）与线上运行（Online）表现不一致”的问题，专家给出了系统性的工程化解答。
离线评估环境下，测试数据集是相对静态与饱和的，模型在有限的分布内能够表现出优异指标；然而一旦模型部署至线上生产环境（Online Serving），面对真实海量用户时必然会遇到离线阶段从未出现过的未知特征（Edge cases 与分布漂移），从而导致线上性能出现明显下滑。解决该断层的唯一路径是构建端到端的数据反馈闭环（Data Feedback Loop）：
- **实时监控线上异常**：捕获线上推理过程中出现的新分布数据与低置信度样本。
- **持续重训与数据同步**：将线上捕获的新数据清洗后反哺至训练集，保证训练数据分布始终与生产环境保持最新同步。
- **往复迭代优化**：将模型演进视为动态的往复循环（Back-and-forth process），通过不间断的数据更新确保模型在线上始终维持高质量服务水平。
### [38:21](https://www.youtube.com/watch?v=UZv87qrKM54&t=2301s) 生产环境与训练集的数据漂移量化方法
提问环节中，现场观众提出了一个核心问题：大家都在谈从生产环境（Production）回流数据到训练流程的反馈闭环，但对于计算机视觉这种高维且丰富的富媒体数据，究竟该如何进行**量化衡量**？观众指出，像分辨率（Resolution）是否有变化这种指标过于肤浅（Trivial），而在表格数据（Tabular Data）中通常有现成的统计学指标，计算机视觉领域是否有对标的量化手段。
针对这一问题，嘉宾 Vijayraj Gohil 解释道，常规做法是在测试数据和生产运行模型之间持续监控准确率（Accuracy）、精确率（Precision）等性能指标，这能初步判断模型在生产环境中的表现与离线测试的差距。但如果模型性能出现下滑，要找出根本原因，则需要采用类似 A/B 测试或在线数据抽样的方法。因为计算机视觉数据具有极高维度的特性，工程上通常会将线上下载的生产数据与既有训练数据投影到**嵌入空间（Embedding Space）**中进行可视化与相似度对比。通过观察嵌入空间中的分布差异，团队能够发现特征偏移的线索，进而确定需要在训练集中补充哪些参数或样本以弥补这一差距。
```mermaid
flowchart LR
accTitle: 生产数据与训练数据的特征漂移检测流程
subgraph DataInputs ["数据输入"]
A["生产在线数据<br/>(Online Data)"]
B["离线训练数据<br/>(Training Data)"]
end
subgraph Analysis ["特征分析与对齐"]
C["嵌入空间投影<br/>(Embedding Space)"]
D["语义匹配 / 特征漂移评估<br/>(Semantic Matching)"]
end
subgraph Action ["闭环反馈"]
E["人工标注补充与<br/>训练集微调迭代"]
end
A --> C
B --> C
C --> D
D --> E
```
另一位嘉宾（Arpita Vats）进一步从语义角度补充，由于数据已经被转换到了嵌入空间，团队可以直接在生产新数据和既有训练数据之间做**语义匹配（Semantic Matching）**。这种方法可以直接评估特征漂移（Feature Drift），量化线上输入与训练基准之间的分布偏移，准确定位导致模型推理行为发生变化的潜在原因。
- **高维可视化**：利用模型提取的特征向量将图像映射到高维嵌入空间，直观展示分布差异。
- **语义匹配与漂移评估**：通过度量嵌入向量之间的距离与相似度，计算新旧数据之间的特征漂移（Feature Drift）。
---
### [40:28](https://www.youtube.com/watch?v=UZv87qrKM54&t=2428s) 仅依赖合成数据训练的挑战与应对策略
随后，有观众提出了两个紧密相关的问题：
1. 如果团队面临极端的数据匮乏，**完全只有合成数据（Synthetic Data）可供训练**，应该采取什么策略来尽可能提高真实度，从而训练出高精度的模型？
2. 从传统的计算机视觉算法向重度依赖机器学习（ML）与大型语言模型（LLMs）的现代 CV 方法过渡有多困难？
Vijayraj Gohil 指出，在现实项目中，“几乎全靠合成数据、真实场景数据极少甚至为零”的情况其实比想象中更为普遍。在这种情况下，使用合成数据的思路不应仅局限于弥合真实与合成的缝隙，更有效的做法是：借助社区中在海量外部数据上预训练好的优秀开源模型，利用手中的合成数据将该模型**微调迁移到自己的特定领域（Domain Adaptation）**。随后，依然需要对比合成数据与偶尔收集到的少量真实数据在嵌入空间中的维度与分布差异，并通过人工标注员（Human Annotators）介入针对性地修正并收窄差距。
Teresa O'Neill 则给出了一个非常直率且关键的建议：“如果你只有合成数据，目前最实在的建议是——**降低你的期望值（Lower your expectations）**。”她强调，当依赖预训练基座加合成数据时，必须将问题范围切得极其狭窄。她举例分享了农业领域的一个真实案例：
- **大豆田特定杂草识别**：类似于之前讨论过的草莓采摘场景，团队需要在一特定农作物（如大豆）田中识别特定类型的杂草。
- **狭窄场景的可行性**：团队成功合成了专门的数据集，用于微调现有的预训练视觉模型，最终该模型在大豆田识别该特定杂草上表现良好，但它“完全无法处理任何其他任务”。
- **假设空间的控制**：如果你的任务假设空间（Hypothesis Space）过于宽泛、维度极高，单纯依靠合成数据几乎注定会失败。
---
### [43:28](https://www.youtube.com/watch?v=UZv87qrKM54&t=2608s) 合成数据落地的三大陷阱与圆桌总结
针对合成数据的工程落地，Ahmed Rashad 总结并提出了三个容易导致次优结果（Sub-optimal Results）的核心避坑指南（Gotchas / Watch-outs）：
1. **时效性与退化风险（Time Dependency）**：即便合成数据在项目初期起到了作用，“今天有效的方法，明天很可能就会失效”。现实世界的数据分布处于动态变化中，团队必须保持持续测试与自适应调整。
2. **测试维度的全面性（Stress Testing on Real Outliers）**：评估模型时，不能仅仅在模型设计好的理想合成场景下做测试。必须使用贴近实际业务场景的真实用例进行严格检验，特别是那些**异常值、极端边缘案例（Outliers & Exceptions）**，以此验证其是否真正解决了实际业务痛点。
3. **合成数据与人工输入的反哺闭环（Human-in-the-loop）**：实际经验中存在一个颇具讽刺意味的现象——很多用于生成合成数据的生成模型，往往在引入了**人工真实数据（Human Data）**反馈后，生成质量才会出现质的飞跃。
Ahmed 强调，对待合成数据必须保持开放心态并随时管理好预期，行业中并不存在一劳永逸的“银弹”（"There's no magic bullet"）。如果某一方法在当前阶段管用就继续使用，一旦失效就必须快速迭代调整。
在全场讨论结束后，主持人向 Arpita Vats、Saurabh Kumar、Vijayraj Gohil、Teresa O'Neill 和 Ahmed Rashad 等全体与会嘉宾致谢，同时也对现场观众提出的高质量深度问题表达了感谢，本次关于构建高可用计算机视觉数据流水线的分论坛圆满结束。
## Notable quotes
> [12:27](https://www.youtube.com/watch?v=UZv87qrKM54&t=747s) 谁在乎最佳实践？如果你正在构建前所未有的创新系统，最佳实践往往根本不适用 — Ahmed Rashad
> [10:05](https://www.youtube.com/watch?v=UZv87qrKM54&t=605s) 万事万物皆有其时其地，绝大多数错误都源于在错误的时间将工具放错了位置 — Ahmed Rashad
> [22:20](https://www.youtube.com/watch?v=UZv87qrKM54&t=1340s) fails so hard and so weird — Teresa O'Neill
> [22:20](https://www.youtube.com/watch?v=UZv87qrKM54&t=1340s) I own the road now — Teresa O'Neill
> [24:32](https://www.youtube.com/watch?v=UZv87qrKM54&t=1472s) more or less cracked — Ahmed Rashad
> [36:47](https://www.youtube.com/watch?v=UZv87qrKM54&t=2207s) 垃圾进，垃圾出（Garbage in, garbage out） — Saurabh Kumar
> [40:28](https://www.youtube.com/watch?v=UZv87qrKM54&t=2428s) 如果你只有合成数据，目前最实在的建议是——降低你的期望值（Lower your expectations） — Teresa O'Neill
> [43:28](https://www.youtube.com/watch?v=UZv87qrKM54&t=2608s) There's no magic bullet — Ahmed Rashad
## Mentioned
Arpita Vats, Saurabh Kumar, Vijayraj Gohil, Teresa O'Neill, Ahmed Rashad, LinkedIn, Meta, Uber, Kraft Foods, 2K Games, NYU, iMerit Inc., Perle AI, Scale AI, Amazon, McKinsey, MIT, Amazon Go, CVPR, Facebook, HIPAA
## Video description
Arpita Vats, Senior AI Engineer, LinkedIn
Saurabh Kumar, Senior Manager, Ex-Meta, Uber, Kraft Foods
Vijayraj Gohil, Machine Learning Engineer, 2K Games
Teresa O'Neill, Director of Solutions Architecture, Natural Language Understanding, iMerit Inc.
Ahmed Rashad, Founder & CEO, Perle
Abstracts
This session will focus on the importance of building robust data pipelines for computer vision projects, with an emphasis on data labeling, management, and strategy. Panelists will discuss the challenges of curating large-scale datasets, ensuring high-quality labeled data, and implementing efficient data workflows to support model training and optimization. Learn how to structure and maintain a strong data pipeline that supports continuous learning and improvement in computer vision models. The discussion will also cover data augmentation, synthetic data, and best practices for handling edge cases and imbalanced datasets.
[Source](https://www.youtube.com/watch?v=UZv87qrKM54)
Source quotes
[12:27] Ahmed Rashad: 谁在乎最佳实践？如果你正在构建前所未有的创新系统，最佳实践往往根本不适用
[10:05] Ahmed Rashad: 万事万物皆有其时其地，绝大多数错误都源于在错误的时间将工具放错了位置
[22:20] Teresa O'Neill: fails so hard and so weird
[22:20] Teresa O'Neill: I own the road now
[24:32] Ahmed Rashad: more or less cracked
[36:47] Saurabh Kumar: 垃圾进，垃圾出（Garbage in, garbage out）
[40:28] Teresa O'Neill: 如果你只有合成数据，目前最实在的建议是——降低你的期望值（Lower your expectations）
[43:28] Ahmed Rashad: There's no magic bullet
Source quotes
[10:05] Teresa O'Neill: turning the right knobs at the right time.
[22:20] Teresa O'Neill: when the model fails, it fails so hard and so weird.
[26:06] Ahmed Rashad: the cost of a failure is significantly higher
[36:47] Saurabh Kumar: garbage in, garbage out
[40:28] Teresa O'Neill: Lower your expectations if all you have is synthetic data.
[43:28] Ahmed Rashad: What's working today very likely will not work tomorrow
[43:28] Ahmed Rashad: If it works, it works, right?... It's not going to be the magic bullet. There's no magic bullet. And even if it's a magic bullet today, tomorrow it won't be. So, just keep experimenting and testing and if it—when it stops working, it stops working. If it works, it works.
Original transcript
[0:04] Thank you so much. Welcome everybody to
[0:06] this session. I would like to give an
[0:08] opportunity to the panelists to
[0:10] introduce themselves one by one before
[0:12] we start diving deeper into the
[0:14] questions that we're going to discuss
[0:15] today in this panel.
[0:18] >> Hello. Hey everyone, my name is Vijay. I
[0:21] currently work as a machine learning
[0:22] engineer at 2K Games. I worked
[0:24] previously at retail tech and machine
[0:26] learning and also work as a machine uh
[0:28] machine learning researcher at NYU.
[0:31] >> Hi everyone, good afternoon. I'm Teresa
[0:34] O'Neal. Um I'm a linguist and kind of
[0:38] data operations specialist uh running
[0:40] the Genai business at Immarit, a company
[0:42] that specializes in data services and AI
[0:46] data transformation for the automotive
[0:48] industry, the medical industry and um
[0:50] broadly uh generative AI and enterprise
[0:52] technology.
[0:55] Hello everyone. So uh I'm Arata. I'm
[0:58] currently a senior engineer at LinkedIn.
[1:00] earlier to that during my PhD during my
[1:04] PhD I did work on computer vision and
[1:07] NLP was my main focus and so I have been
[1:10] working in the field of LLM with um
[1:13] Genai for last 3 four years now and
[1:16] that's about me
[1:18] >> uh hello everyone I'm Ahmed Rashad the
[1:21] founder of Pearl AI uh before that I
[1:24] built and led human operation human data
[1:27] operations at scale AI from 2018 into uh
[1:30] up until a couple years ago uh was
[1:33] building AI apps at Amazon before that
[1:36] and McKenzie and uh was a MIT studying
[1:40] ops research for a long time. Uh but
[1:42] actually interesting I start my career
[1:44] as an offshore driller before going into
[1:45] AI.
[1:46] >> Thank you everyone for introducing
[1:48] yourselves. So I'd like to start our
[1:50] discussion with the first question and I
[1:53] would love to get your perspective on
[1:55] how do you think are computer vision
[1:58] pipelines different from the traditional
[2:01] data pipelines that all of that most of
[2:04] us are more used to. Um yeah sure. So I
[2:09] think uh for computer vision uh it's
[2:12] different from the rest of the data
[2:13] pipeline because if you compare like NLP
[2:15] data pipeline it's more about like
[2:17] either it's just a text base or just a
[2:19] audio based which you can convert into
[2:21] uh any um text base format but I think
[2:24] when it comes to vision it has a lot of
[2:27] perspective like it can be combination
[2:29] of uh video like um images plus audio
[2:32] images plus um um like it also needs to
[2:35] have the um I mean the time stamps right
[2:38] it can't be like if you're taking a
[2:39] video completely it needs to include the
[2:42] subtitles and all those information in
[2:43] the same data pipeline. So I think for
[2:45] vision the data pipeline needs to be
[2:47] much more better as compared to the any
[2:50] other um data pipeline if that's what u
[2:52] I think answers that question because I
[2:55] think uh for computer vision it's the
[2:57] pipeline needs to be much more rigorous
[2:58] as compared to any other um platforms
[3:01] for data like text or um audio based.
[3:06] >> Got it. Thank you. with that.
[3:07] >> I have a followup to that. I think that
[3:09] there's another kind of interesting
[3:10] difference. Um we think about kind of
[3:12] the differences between um LLM oriented
[3:15] data pipelines and CV. There there is
[3:17] kind of a philosophical difference. In
[3:20] computer vision, you're trying to model
[3:21] the actual real world. It's knowable.
[3:24] There is a ground truth of what the
[3:26] objects are, where they're located, and
[3:28] how they're moving and how they're
[3:29] behaving. with LLM driven pipelines,
[3:32] you're trying to model a cognitive
[3:33] reality or a social reality where there
[3:35] might not necessarily be a ground truth
[3:37] that you can actually measure. So it
[3:39] means that you you can approach quality
[3:41] control and definitions of accuracy in
[3:44] potentially a much more objective way
[3:46] and you can bring a lot of rigor uh to
[3:48] CV pipelines that very quickly becomes
[3:51] subjective and difficult to pin down for
[3:54] other types of pipelines. there's a lot
[3:56] of opportunity for rigor in a quality
[3:58] control in a requirements design process
[4:01] in CV.
[4:02] >> Thank you. So as a followup to that I
[4:05] was wondering what was the most
[4:07] failureprone
[4:08] link in a computer vision data pipeline
[4:11] today. So for instance you have
[4:13] ingestion labeling versioning
[4:16] monitoring. So what do you think is the
[4:18] most failureprone and what are the
[4:20] guardrails that can be used to prevent
[4:23] those failures in computer vision
[4:25] pipelines?
[4:27] So one thing that comes on top of my
[4:29] mind it's like I think the quality of
[4:31] the data is very much uh something that
[4:34] could be uh like uh from different
[4:37] distribution or the quality of the data
[4:38] or the pixel value of the data is much
[4:40] more important the vision if especially
[4:42] we're considering the images it has for
[4:44] if model to there could be some noise in
[4:47] model that's still doable but if the
[4:48] there's noise in your data if there's
[4:50] noise in your images itself it's hard
[4:52] for a model to actually make any sense
[4:54] out of those uh data so I think The
[4:56] quality of the data is the most
[4:57] important part for any computer vision
[4:59] pipeline. And then obviously the
[5:01] versioning like as we see more and more
[5:02] things coming up, we need to update
[5:04] those um data that we're actually
[5:06] inputting the models to modify because
[5:09] otherwise our distribution would be much
[5:11] more older compared to what the model is
[5:12] actually seeing in the real world when
[5:14] we are uh either serving it or when we
[5:16] are doing it in the production and the
[5:17] real world applications. So I think two
[5:20] of the things that comes on top of my
[5:21] mind is definitely the data quality and
[5:23] how well we version our data with the
[5:25] recent uh data that we have right now.
[5:29] >> Following up on that just just thinking
[5:31] about a lot of the pipelines that we've
[5:33] seen in the past and any of those any of
[5:36] those points can have failures
[5:38] throughout. Of course the question is do
[5:40] you have the right data? Do you have the
[5:41] right amount of data? Is it processed
[5:42] properly? Are the uh are you adding the
[5:45] right insights to them and are you
[5:47] ingesting them correctly? And all of
[5:48] these are great. The the real problem,
[5:50] the most difficult part where most
[5:52] failure happens isn't necessarily where
[5:53] the failure happens. It is in
[5:55] identifying what actually caused the
[5:57] failure. You can very often confuse bad
[6:00] labeling with having the wrong data in
[6:02] the first place or not having enough
[6:03] data or or ingesting it at the wrong
[6:05] time even. So the in my experience the
[6:09] most challenging part hasn't been
[6:10] necessarily of course humans are
[6:11] incredibly variable and so on. So that
[6:14] naturally just imposes more failure
[6:15] points. But that said, getting to the
[6:19] root cause of why the pipeline failed,
[6:21] that is the most challenging piece that
[6:23] at least I've seen.
[6:26] >> Thank you, Ahmed. Um, as a follow-up
[6:29] question, I would love to get your
[6:31] perspective. If you think of your of a
[6:34] recent project that you've worked on,
[6:36] what were the exact controls that helped
[6:40] you um dial things up or down? So for
[6:43] instance if you look at turnaround SLA
[6:47] the QA protocol or the cost or asset and
[6:51] what has moved the needle most right um
[6:54] and when you think of comparing the
[6:57] performance of pipelines where there
[6:59] were pre-labeling involved with models
[7:02] versus humans labeled the data set. Um
[7:06] how do you think of that relative
[7:08] performance? So looking looking back at
[7:11] that as well, it's um the there's often
[7:14] a fallacy where we where and we see this
[7:17] with a lot of our customers where they
[7:19] assume that one thing is just the
[7:21] absolute ultimate absolute answer to
[7:23] everything. So oh synthetic data
[7:25] synthetic data is better. Well,
[7:26] synthetic data is better in certain
[7:28] areas in some use cases and not constant
[7:31] over time. Over time that will change.
[7:33] So the the the right answer is is almost
[7:36] always it depends on where you are in
[7:39] the development stages and the right
[7:41] answer is do you have the right in
[7:43] system built together in place that
[7:46] actually addresses the needs your needs
[7:48] at that stage. If you're building a
[7:50] golden data set you're going to go very
[7:51] very heavily towards humans with maybe
[7:53] with some machine intervention. If
[7:56] you're in mass production or you're
[7:58] pre-training, you are going to go very
[8:00] very heavily on machine labeling and
[8:03] very very light on human interaction.
[8:04] So, so the answer severely varies
[8:07] depending on where you are, what are you
[8:08] trying to build and what are the
[8:10] previous results. Remember, we haven't
[8:11] fully cracked how this actually works.
[8:13] We we know how to manipulate, but we
[8:15] haven't fully cracked it. So being very
[8:17] very precise with how we turn the knobs
[8:19] is not something which we fine- tuned
[8:21] yet but we know roughly where we need to
[8:24] go. Right.
[8:26] >> Follow up on following up on that. I
[8:27] think the problem mostly that I face is
[8:29] like data labeling and schema evolution
[8:32] and that has been a major problem for me
[8:34] as well and that's the general problem
[8:36] that the industry has faced for a long
[8:38] time and I feel thing that worked for me
[8:41] is uh having data contracts and having
[8:43] drift monitors to monitor like for every
[8:46] different data that you have trained the
[8:47] model with you also have a drift monitor
[8:50] to see how the model performance has
[8:52] been drifting and for that you also have
[8:54] a lineage force promotion on that to see
[8:56] like for a specific data and the model a
[8:59] certain performance of that if you can't
[9:02] really trace back on what data is making
[9:04] it perform like that it doesn't go to
[9:06] the production so that's my take on that
[9:09] >> thank you Vijay and Ahmed for your
[9:11] perspectives I had a follow-up question
[9:13] which either of you can answer and the
[9:16] question is that name one task where
[9:18] synthetic or augmentation truly closed a
[9:22] gap and one where it failed right so for
[9:24] example you can think of a longtail edge
[9:27] case where it failed and how did you
[9:30] validate the reality gap and avoided
[9:33] overfitting to the generator.
[9:35] >> Um there's a lot of of course a lot of
[9:37] hype around synthetic data being able to
[9:40] solve a lot of these problems. It's not
[9:41] the the panacea that people kind of
[9:43] think it it could be. It has there
[9:46] really has to be a deliberate design
[9:47] cycle. Why are we trying to synthesize
[9:49] this data? What is the gap that it's
[9:51] trying to address and does it make sense
[9:52] to use this tool to address that gap? So
[9:55] in one context we found that the the
[9:58] answer was that synthetic data could be
[9:59] very helpful but it still required human
[10:01] oversight. That context for us was in a
[10:03] healthcare use case where data on
[10:05] availability and privacy issues limited
[10:08] the availability of of real in the wild
[10:12] data. So we were able to synthesize um
[10:15] dialogue interactions simulating
[10:17] experiences uh between clinicians and
[10:20] patients across a number of different
[10:22] medical subsp specialties in a number of
[10:23] different languages to create a data set
[10:25] that could help not in computer vision
[10:27] but it's related right this was in an an
[10:29] LLM application and the performance was
[10:33] pretty good but it still required
[10:34] extensive human monitoring and the
[10:36] reason we chose that problem to address
[10:38] with synthetic data was precisely
[10:40] because it's a known gap in real data,
[10:43] it it eliminates a lot of the concerns
[10:45] about privacy and um and HIPPA and the
[10:50] stakes are a bit lower that things will
[10:52] go wrong if you have the right kind of
[10:53] oversight. So, it's all about I think
[10:54] choosing the use case for synthetic data
[10:57] very deliberately, addressing a real gap
[11:00] and applying it at the right sort of
[11:02] part of the model development life cycle
[11:04] where you're managing those risks to
[11:06] accuracy to safety and to cost in a
[11:09] sensible way. As Ahmed said, it's
[11:11] turning the right knobs at the right
[11:12] time and recognizing what those
[11:14] trade-offs are. So, synthetic data can
[11:16] be a great tool, but it's not everything
[11:19] and it needs to be handled with care. It
[11:21] needs to be part of a really deliberate
[11:23] design process.
[11:25] >> I'd love to second that. I I
[11:28] wholeheartedly agree. Everything has its
[11:30] place. Everything has its time and it
[11:31] needs to be put in the right place. Most
[11:33] errors happen when you don't put things
[11:34] in the right place at the right time.
[11:36] and follow up on your your examples like
[11:38] we've we've seen exactly the same thing.
[11:40] Synthetic data is great at getting
[11:42] things started. Maybe develop a PC and
[11:44] and and okay, fantastic. But it also
[11:46] generates data that's more or less
[11:48] predictable and not 100% reflective of
[11:51] what the reality is. You go into fields
[11:53] where we haven't ingested the entirety
[11:55] of human knowledge in like medical like
[11:57] you were mentioning or legal and so on
[11:59] and so forth where a lot of the info is
[12:00] still not properly ingested and all of a
[12:02] sudden you realize you have all sorts of
[12:04] edge cases that have no place in reality
[12:06] and you're missing the actual edge
[12:08] cases. How do you solve it? For example,
[12:10] in a medical use case where let's say
[12:12] you're you're trying to get in input
[12:14] from multiple doctors and synthetic data
[12:16] is producing this predetermined thing
[12:18] while the doctors themselves don't know
[12:19] what the real answer is. So you'll never
[12:21] get the case like that from synthetic
[12:23] data. It it's just not happening. We
[12:24] haven't ingested it yet. We don't know
[12:26] it exists.
[12:27] >> Totally. Thank you Ahmed and Teresa for
[12:31] your perspectives. Going back to my uh
[12:35] previous question where we were
[12:36] discussing the various components of a
[12:41] computer vision data pipeline.
[12:44] I would love to understand what is your
[12:46] versioning and lineage contract. So for
[12:49] instance, data sets, labels, models and
[12:54] the comment that Ahmed made around
[12:58] finding the bug is so important when a
[13:00] CV pipeline breaks. So if you find a bug
[13:05] uh in a label schema for instance, then
[13:07] how quickly can you reconstruct you know
[13:10] which model saw what when and then sort
[13:13] of work from there to fix things. So I'd
[13:17] love to get your your take on that.
[13:20] Sure.
[13:20] >> Uh I can get this started. So So you you
[13:24] start by doing things deliberately with
[13:27] purpose. So you're not just labeling and
[13:29] mass or you're not just um cleaning up
[13:31] the data in a certain way just because
[13:33] just because it's best practice. Best
[13:35] pract who cares about best practice,
[13:36] right? You're if you're building
[13:38] something novel, then probably best
[13:39] practices don't apply to you anyways. So
[13:42] might as well figure out think of it
[13:44] with a scientific method point of view.
[13:47] So I am doing this I expect this
[13:49] outcome. This is the expected value. It
[13:52] does not come like that. It's okay to
[13:54] pause for a little bit and backtrack and
[13:56] figure out what went wrong. And it's
[13:58] usually one of the few things. It's
[14:00] usually something with the with the
[14:01] machine itself. Sometimes it's something
[14:03] with the human. Sometimes it's something
[14:04] with the process. You got to go back and
[14:05] figure what what's going on there. And
[14:08] you go back and figure what's going on.
[14:09] You adjust it. You think you've got it.
[14:11] You adjust it. You see the output. The
[14:13] output. Has the output changed? Yes. No.
[14:15] Yes. Okay. We're on the right path. No.
[14:17] I did not find the root cause. Time to
[14:19] go back. And you need to do that before
[14:21] you actually progress. I understand
[14:22] timelines, deadlines, all of that. But
[14:24] to to the point here is that rapid
[14:26] iteration is the key versus mass volume
[14:30] just for the sake of mass volume. So,
[14:33] you need to work. You need to to be
[14:34] ready to iterate every day. I'm I'm not
[14:37] even going to say every every week.
[14:38] Every week is probably too late. You
[14:40] need to be able to iterate on this every
[14:42] day because you need to do hundreds if
[14:44] not thousands of micro tweaks along the
[14:46] way. And that's what we've seen the
[14:48] difference between really really
[14:49] successful builders and others.
[14:52] >> Yeah, I think to add on top of it um I
[14:55] kind of agree but I kind of not agree as
[14:57] well. So I think uh it's obviously good
[15:00] to first do the whole thing and then
[15:02] come back and see where it's lagging or
[15:04] what's causing that failure that what
[15:06] you expected. But I think there are
[15:07] multiple phases in a pipeline, right? So
[15:09] you don't have to wait for the complete
[15:11] pipeline to finish it rather you can
[15:14] finish one part of the pipeline and you
[15:15] know what is expected output of this
[15:17] pipeline and if you're not seeing that
[15:19] output at that particular pipeline that
[15:21] means something is wrong here rather
[15:23] than you finishing the whole pipeline
[15:24] and then figuring out like okay this was
[15:26] wrong and maybe I can backtrack from the
[15:28] beginning rather we can do it like small
[15:30] steps may might be it might not turn
[15:34] around such far as fast as you expected
[15:36] but you might have the result which you
[15:38] were kind of looking forward to. So
[15:40] maybe rather than a week it might take
[15:42] you two weeks but uh at the end you
[15:43] might get the results uh which won't let
[15:46] you to backtrack to figure out like
[15:47] which step of your pipeline actually
[15:49] caused that issue. So I think it's also
[15:51] important like for each and every track
[15:53] you do it like a pipeline you do it you
[15:56] make sure that the output of this
[15:58] pipeline is expected as was you were uh
[16:00] assuming it to be the distribution of
[16:02] the data or all other things that you uh
[16:04] expect in the pipeline to come out which
[16:06] contribute as your final output from the
[16:08] complete pipeline
[16:12] >> on that I think uh when you're trying to
[16:15] find a bug on whether the data has been
[16:16] corrupted or not some other theoretical
[16:19] ways could be like how do you try to
[16:21] find the automation tools and validate
[16:23] them on how they were created. So that
[16:25] can give you a certain direction on what
[16:27] kind of problems they might have
[16:28] introduced. If it's humans, you can have
[16:30] a set of validation questions to also
[16:32] see like what problems they might have,
[16:34] what assumption they had before starting
[16:36] on annotating this data. So that can
[16:38] lead us to a certain direction to see
[16:40] like what problems that have introduced.
[16:43] >> I think another follow-up to that is
[16:45] just this key word here management. I
[16:47] think bringing a lot of practices from
[16:50] traditional kind of manufacturing and
[16:52] services and then the agility that we
[16:55] need for these daily type iteration
[16:56] cycles and a really good partnership for
[16:58] prototyping and R&D and finding the
[17:01] balance there is key because you can get
[17:03] a really great early warning system from
[17:05] a labeling operations partner. For
[17:07] example, if there's something weird with
[17:09] this the source data that just got
[17:11] missed and there was maybe a minimal
[17:13] kind of triage or curation process. We
[17:15] see this all the time. and you have a
[17:16] partner that comes in and here you go.
[17:18] Here's your batch. Go for it. We're
[17:19] excited for the turnaround. And we look
[17:21] at it and we're like, "Oh boy, you
[17:23] didn't realize, right, that this was
[17:24] collected during a festival and like the
[17:27] driving behavior is completely different
[17:29] than normal or it was collected during a
[17:30] really weird storm and we didn't solve
[17:33] for the impact of people driving around
[17:36] floods or whatever it is, right? Um, you
[17:39] can get that early warning signal from a
[17:41] partner that's really keyed into that.
[17:43] you have really good indexing on your
[17:44] batches and your cycles. So you know you
[17:46] can trace the lineage of of the whole
[17:48] data through that cycle and you don't
[17:50] need to wait to see something weird in
[17:51] model performance later. You get the
[17:53] signal right away when the um the data
[17:56] is consumed for annotation and analysis.
[17:58] Some of that can be automated but a lot
[18:00] of it really does just come from organic
[18:01] kind of human eyeballs right you look at
[18:04] it and you say this isn't what we
[18:05] thought it was. Um and similarly in
[18:07] managing annotation schema and quality
[18:10] frameworks, you have a really robust
[18:12] auditing program where that you've got
[18:13] independent reliable reporting on
[18:16] quality audits both for the raw data and
[18:18] for the annotation. You can catch a lot
[18:21] in that cycle as well from the support
[18:24] of a labeling ops partner where you
[18:26] don't have to necessarily expend all the
[18:27] resources there. But again, you're
[18:29] getting that signal and that opportunity
[18:31] to course correct before you've spent a
[18:33] bunch of money on compute or put
[18:34] something out and seen, you know,
[18:36] drifter or problems in production. You
[18:38] can get it much earlier. So, from our
[18:40] perspective, kind of working on that
[18:41] early, you know, pre-training and early
[18:44] post-training and um reinforcement
[18:46] learning cycle, we love the early
[18:48] warning. We that's something that we're
[18:49] excited to always offer to to partners
[18:51] so we don't have to wait until after the
[18:53] developers have consumed the data.
[18:56] >> Thank you, Teresa. I would I would open
[18:59] up um this question to all of you and
[19:02] would love to know an example from a
[19:04] practical standpoint.
[19:06] What was the purpose for which you were
[19:08] building a computer vision pipeline and
[19:10] what was one of the more frequent use
[19:13] cases for which you had to build a
[19:15] computer vision pipeline?
[19:21] So I don't think I have currently that
[19:24] I'm working on but I think during my
[19:26] research days I did work on some of the
[19:28] tasks that I had to build. So basically
[19:29] we were developing u more kind of um
[19:32] automated retail checkout and I think I
[19:35] don't know if people are aware with
[19:36] Amazon Go they kind of do the same
[19:38] thing. Uh however the difference was
[19:40] like um the user are still picking up
[19:42] the objects from the stores but we don't
[19:44] have any person checking out for them
[19:46] instead of a selfch checkout. So I think
[19:48] one of the challenges that I was having
[19:50] there is like there's a lot of things in
[19:51] the surrounding when the user comes to
[19:53] the um cell checkout. They probably keep
[19:56] all their stuff there and it's hard for
[19:58] the like camera to detect one object at
[20:00] a time and then there are a lot of
[20:02] noises around you like your hands and
[20:04] all the other things. So I think uh uh
[20:06] we basically created a n painting
[20:08] technology where we uh not technology
[20:10] like n painting uh algorithm where we
[20:13] actually um make sure that we remove all
[20:15] the noise and make sure that that object
[20:17] is a 3D object but the model is mostly
[20:19] trained on the 2D object. So the model
[20:21] should be able to understand like the
[20:23] different angles of the object and
[20:25] identify what particular this object is.
[20:27] So I think one of the challenge there
[20:28] was to get rid of all the noises the
[20:30] hands because mostly when user are
[20:33] scanning the object they have their
[20:34] hands on it. So it's hard for you to
[20:37] remove that hand and still have the
[20:39] object ID intact so that model can
[20:41] recognize it. So that was some one of
[20:44] the challenge that I faced and we were
[20:45] able to probably get the all the noises
[20:48] around it and still model was able to
[20:50] detect those. uh and I think one of the
[20:52] key factor for the model to able to
[20:54] detect those object clearly was uh
[20:57] probably because of the data that it was
[20:58] trained on. So it had all the different
[21:00] angles of the same object. So even
[21:02] though the object might have some
[21:04] overlap on the top of the hand and it
[21:06] was removed but it was still able to
[21:08] understand like what's specific based on
[21:10] the contrast of the object and other
[21:11] different things for that specific
[21:13] object. So I think for me um I've been
[21:16] worked in I I've been working in
[21:18] industry as well as in research for
[21:20] quite a while now. So for me it's been
[21:22] like a two-part problem. For industry
[21:25] specifically it's more about utilizing
[21:27] the existing foundational models and
[21:29] then kind of fine-tuning retaining that
[21:31] for your own domain and then mixing up.
[21:33] It could be like multiple different
[21:34] models image segmentation object
[21:36] detection and then trying to build up a
[21:38] problem uh build up a solution to a
[21:40] certain problem. And then there's
[21:42] another problem when I'm going to
[21:43] research is like how do you actually
[21:46] improve the existing foundation models
[21:48] capabilities like just the raw power of
[21:50] its segmentation and u general
[21:52] detection. So it all depends on like
[21:55] what kind of problem you are working in.
[21:56] So in industry it's more like um
[21:59] engineering challenges infra and like
[22:01] the amount of data that you're going to
[22:02] process and also like in during the
[22:04] training uh during the testing phases
[22:07] how do you online sample and how do you
[22:09] monitor this model performance? but more
[22:11] in like a research setting. It's more
[22:14] about which exact data has been
[22:17] contributing to model certain
[22:18] capabilities.
[22:20] >> One uh exciting kind of use case that
[22:22] that we've spent a lot of time in at
[22:23] IMEIT with a focus and kind of an origin
[22:26] story anchored in autonomous mobility is
[22:29] kind of a shift toward a multimodal
[22:31] approach and a focus on trust and
[22:33] explanability. So object detection
[22:36] models and predictive models and
[22:38] behavior models are maturing. Um but
[22:40] when the model fails, it fails so hard
[22:43] and so weird, right? Like why is it
[22:45] doing what it's doing? Um and there's
[22:48] obviously a desire, you know, in the
[22:50] interest of safety and trust to be able
[22:53] to explain the behavior. And so some of
[22:55] our customers are taking the approach of
[22:56] embedding or or kind of um hybridizing
[22:59] these these models with language and
[23:01] reasoning models where they can
[23:02] verbalize and use human linguistic
[23:05] communication to say I am driving the
[23:07] car down the street and I am observing
[23:09] that there is a pedestrian who looks
[23:10] like they may have the intent to
[23:12] jaywalk. I am making a choice to slow
[23:14] down. So when you can hear the talk
[23:16] track of the model's behavioral
[23:19] decision-m, it gives you a little bit
[23:21] more of a window into why it's making
[23:23] the choices that it's making, which can
[23:26] increase the trust and it also gives you
[23:28] um access to kind of analyze what's
[23:30] going on inside that black box when
[23:32] things break. So, for example, if it's
[23:34] narrating that it thinks there's a
[23:35] pedestrian with intent to cross and it's
[23:38] misprocessed, you know, the reflection
[23:39] of somebody in a window or there's a
[23:42] really unusual animal and it thinks that
[23:45] maybe it's a a dog on a leash, but it's
[23:47] in fact a peacock and we're in Miami and
[23:49] there's like peacocks just go in the
[23:51] road and stop and other animals
[23:53] typically will run away, right? Not in
[23:55] Miami. The peacocks go in front of the
[23:57] car and they just are like, I own the
[23:58] road now. So you have to handle that
[24:00] situation a little bit differently than
[24:02] you might if it's just another bird. So
[24:05] having this narration and this talk
[24:06] track to help you make decisions about
[24:08] what failure cases and what risk cases
[24:11] you want to go in and um analyze more
[24:14] deeply or run some additional test
[24:16] scenarios against or identify an area of
[24:18] risk that may need shoring up with
[24:20] additional reinforcement learning is I
[24:22] think a really powerful application.
[24:24] where we're seeing this move toward more
[24:26] multimodality in our models and
[24:28] different tools that can be used to
[24:30] enhance the the safety.
[24:32] >> Emma, do you want to weigh in on that?
[24:34] >> Yeah, sure. So, boy, where where do I
[24:36] start with computer vision uh models?
[24:38] Well, they've definitely gone through
[24:40] significant evolution. So, of course,
[24:43] one of the more key use cases was
[24:45] autonomous vehicles, self-driving cars,
[24:47] drones, delivery robots, you name it.
[24:50] But more and more to Theresa's point is
[24:52] that we're seeing it more evolve into
[24:55] multimodal and in more complex
[24:57] applications where you need sometimes
[24:59] you need a lot more judgment contextual
[25:01] judgment based on that. So so um robots
[25:04] that help around the house or industrial
[25:06] robots that's that's that's very very
[25:08] exciting. Uh one one of the very
[25:10] interesting use cases is robots that are
[25:12] picking strawberries for example. So you
[25:14] have we have a lot of these examples and
[25:16] there's picking strawberries based on
[25:17] how ripe they are and are they the right
[25:19] level of ripeness and the robots's got
[25:20] to make a decision and then pick and
[25:22] pack and and then it can get feedback.
[25:24] The beauty about it is it can get it can
[25:26] get feedback that oh you picked that
[25:27] specific strawberry it wasn't right and
[25:29] here's why and it can learn but it
[25:31] cannot learn that without actually
[25:33] having inputs the proper inputs from
[25:36] combining language models to into the
[25:38] vision model itself. So it's a it's a
[25:40] very interesting very interesting new
[25:42] pro prog p piece of progress that's been
[25:44] made and um yeah multimodal is is the
[25:47] way forward. More complex applications
[25:49] are happening every day. Autonomous
[25:51] vehicles are I think are more or less
[25:53] cracked. Of course there's a lot more
[25:55] work to be done but it's more or less
[25:56] cracked. And uh I think um robots are
[26:00] probably going to be one of the use
[26:01] cases you see a lot more of in the
[26:04] future.
[26:06] >> Thank you. As a followup, I would love
[26:09] to get your perspective on when you
[26:10] think of multimodel, meaning combining
[26:13] vision, language, tabular data in
[26:16] production. What are the frequent
[26:19] failure modes that are appearing in data
[26:21] collection, indexing, explanability, and
[26:25] have you had success using large
[26:28] language models to autoindex or route
[26:32] the assets to generate human readable
[26:35] rationals?
[26:36] uh to understand the problem and
[26:39] diagnose it.
[26:42] >> So it goes back to the to the to to the
[26:45] core. We have to take it back to the
[26:46] basics where where where are the failure
[26:48] modes? Well, first of all, you might not
[26:49] have the right data. So you got to go
[26:52] start with that. U the second problem is
[26:54] you don't have enough of the right data.
[26:56] they try to augment it with synthetic or
[26:58] with other sources of data and they're
[27:00] not the right data and it actually pulls
[27:02] you back because especially earlier on
[27:05] uh the cost of a of a failure is
[27:07] significantly higher because you don't
[27:09] have that much the model isn't as built
[27:11] so every mistake counts for for 10 or
[27:14] like it's just on a log scale so so
[27:17] earlier on finding the right data and
[27:19] finding the right amount of data I think
[27:20] is the most common failure mode and then
[27:23] after that part of making it the right
[27:26] data. Sometimes you don't have the right
[27:27] data, but you can make it the right data
[27:29] with annotation, with labeling, with
[27:30] eval, with a a lot of automated and
[27:32] manual and human intervention and
[27:34] non-human intervention and so on and so
[27:35] forth. And that part is inherently
[27:39] problematic because you have to deal
[27:41] with large numbers of humans who need to
[27:42] do consistent work over a prolonged
[27:44] period of time. And us as humans, like
[27:46] think about each and every one of us, do
[27:48] you wake up every day at exactly the
[27:50] same time in exactly the same mood?
[27:52] Quite unlikely, right? always a couple
[27:55] of minutes off. But think about it when
[27:57] you tie it back to how sensitive the
[27:59] models are in the earlier stages. That
[28:01] minute or two doesn't have impact on
[28:03] your day-to-day. But if it's incredibly
[28:06] sensitive, then yes, it can have an
[28:08] impact. So having the right humans in
[28:11] the right place in the right system
[28:13] early on is just as critical as having
[28:16] the right data because without it, you
[28:17] don't have the right data no matter what
[28:18] you collect.
[28:20] I think a follow up on that is like um
[28:22] if you target any of the top conferences
[28:24] that's ongoing like CBPR and computer
[28:26] vision specific you will see like most
[28:28] of the research now are based on LLM
[28:31] like how better they can use LLM to
[28:33] describe certain scenario or even
[28:35] autonomous drive I think LLM is being
[28:37] used as a very high scale but I think
[28:40] one thing is pretty vague right now is
[28:42] like we still are trying to understand
[28:44] what's actually happening inside the LM
[28:46] it's not completely undiscovered yet
[28:48] it's kind of still used as black box
[28:50] like okay I give you the scenario now
[28:51] can you describe me what's the chances
[28:53] of this person getting into an accident
[28:55] or based on the autonomous driving
[28:56] example so I feel like yeah it's been
[29:00] growing pretty good and it's making the
[29:02] life easier because now you don't have
[29:04] to train one model for like all the
[29:07] annotations and like um so we have for
[29:09] video like you don't have to train at
[29:11] different models like one for images and
[29:12] then for audio and for all other um
[29:14] different multimodalities so you just
[29:16] can use one model for it but I think it
[29:18] comes with a cost like how well your
[29:21] model is able to understand it and how
[29:22] well you can debug it if something is
[29:24] going wrong because I think it's still a
[29:25] kind of uh under the umbrella of LLM. So
[29:28] you don't have a 100% control on what
[29:31] exactly is happening inside it for
[29:33] certain output that's been coming out of
[29:34] it. So I think yeah multimmorality is
[29:37] great like it's making our life easier
[29:38] so why not but at what cost like are we
[29:41] able to figure out like what exactly is
[29:43] causing and what are the pain points and
[29:44] how can we improve those
[29:46] >> follow up on that I feel like the
[29:48] failure modes that we have seen in the
[29:50] industry has evolved just how the data
[29:53] annotation has evolved in the industry
[29:55] like initially we had custom large
[29:57] models which are kind of trained for one
[29:59] specific problem to having a generic uh
[30:02] language semantic models which can just
[30:04] work on a prompt and then can be used
[30:06] for generally for all different problems
[30:07] that we have right now. So with that new
[30:10] problems also come in like what kind of
[30:12] prompt that you're using in to do this
[30:14] kind of uh annotation and if you change
[30:16] the prompts that you're using how the
[30:19] data quality annotation the initial
[30:20] prelabels that you are getting how what
[30:22] is the quality of that and what's the
[30:24] PCK or the percentage of correct points
[30:27] that you are getting from that so that
[30:29] are kind of the new problems that are
[30:30] coming and the validators or evaluators
[30:33] for those kind of problems and having a
[30:36] variety of prompts it's kind of like
[30:38] still developing right Now, so those are
[30:39] the new challenges I feel needs to be
[30:41] addressed.
[30:43] >> Yeah, I think that a focus on really
[30:46] good evaluators is going to be key
[30:48] because if not, then this explanability
[30:50] problem is going to get a lot worse
[30:51] before it gets better. You have multiple
[30:54] models talking to each other to try to
[30:56] index each other's systems and interact
[30:58] and pass data back and forth. They're we
[31:01] already see, you know, they're coming up
[31:02] with non natural language based
[31:05] communication systems that are really
[31:07] effective that that sort of streamline
[31:08] the communication. They're not legible
[31:10] to us. That's going to get worse. So,
[31:12] it's going to get even more opaque, I
[31:14] think, unless we have a really robust
[31:16] and trustworthy eval system. So, mix of
[31:19] human evals and auto validators and all
[31:22] of these new new kind of emergent tools
[31:24] are going to be really essential. I
[31:26] think as the multimodality gets more
[31:27] complex there's a lot of risk there
[31:29] that's maybe coming soon.
[31:32] >> Thank you for all of your perspectives
[31:34] on that. I know money is an object and
[31:37] especially for the big seven tech
[31:39] companies uh who have spent I believe
[31:42] they've spent 100 billion dollars plus
[31:44] in the last quarter alone on various uh
[31:47] efforts to um to bring AI into the
[31:52] mainstream and uh they are
[31:54] single-handedly acting as a stimulus to
[31:57] the US economy while there's lot of
[31:59] other factors that are pulling us down.
[32:03] So when you think of your um own teams
[32:06] and own companies um what sort of a
[32:09] budget do you set from a financial
[32:12] perspective for your models and uh given
[32:16] the realities of your pipelines? So
[32:18] whether it's with respect to the
[32:19] parameters, latency or cost and how do
[32:23] you decide when you should distill prune
[32:26] or split models by scenario how do you
[32:28] think about that?
[32:30] >> Um yeah I can take it. So I think um
[32:33] from the cost perspective um there's two
[32:36] perspective actually. So when you are
[32:37] serving it to the audience you on the
[32:39] customers you still have that uh issue
[32:42] like how much can you actually use L&Ms
[32:44] to sort of like for example you take
[32:46] Facebook or LinkedIn or any other
[32:48] sources they have like a billion of c
[32:49] users who are using it actively. So in
[32:52] that case you have a limited amount of
[32:54] GPUs that you want to use but then are
[32:56] you able to serve those people at that
[32:59] limited amount. So I think if we are
[33:01] going to use LLM at such a large scale,
[33:03] we definitely need to have much more
[33:04] infra and that comes with a cost and I
[33:07] think that's the point where the
[33:09] industry is more deviating toward using
[33:11] LLM in the offline part of it and then
[33:14] probably using some knowledge
[33:15] distillation technique to use it in the
[33:16] online serving part. So you still save
[33:18] that cost of ENRA but you still have the
[33:21] same performance that uh you are getting
[33:23] from using NLM. So right now if you see
[33:25] most of the model that are serving
[33:27] online app if they're using LM it's only
[33:29] serving like 1 to 2% of the uh customers
[33:32] who are using those but if they want to
[33:34] serve it to the whole 100% they want
[33:36] they will have to do the knowledge is
[33:37] selection for those models to still have
[33:39] those main entities to make sure the
[33:42] performance is still there but it's
[33:44] still not increasing the uh infra cause
[33:46] that they currently have. So I think if
[33:48] we are going to increase the um
[33:51] productivity by productivity I mean by
[33:53] increasing the performance we definitely
[33:54] it comes with a price of uh the amount
[33:57] of GPUs and the cost that uh you need to
[34:00] cover for having that high quality
[34:02] performance. So I think that's when the
[34:03] knowledge distillation plays an
[34:04] important technique because it's uh I
[34:08] think it still able to learn a lot of
[34:10] main factors from the LM and still give
[34:12] you the um equal equal results as an
[34:14] output and with less resources.
[34:17] >> I think for me it's like u the I mean I
[34:20] don't set the budget for the certain
[34:22] project that I work on. So for me it's
[34:24] probably more like how fast that the
[34:26] model that you can run what is the
[34:28] maximum accuracy and the AP that you can
[34:30] achieve through that. So for us it's
[34:32] like we set a certain target over the um
[34:34] precision that we are looking for and
[34:36] once we know that that this is the
[34:38] precision that we want we try to uh
[34:40] start experimentation with really small
[34:42] models and then really large models and
[34:44] then we see like where exactly the small
[34:46] and the larger models are kind of
[34:47] performing gives us a certain benchmark
[34:50] and through that we kind of keeping a
[34:52] certain uh binary search on to like if
[34:54] we half the size of the model what is
[34:56] the performance going to drop and then
[34:58] this kind of search also gives us an um
[35:00] a a lot of explanation on like how the
[35:03] model performance is going to be for if
[35:05] we set a threshold of 1 to 10 and then
[35:07] we try to look for five to seven and
[35:09] things like that. So this really helps
[35:11] us with uh a faster experimentation and
[35:14] um overall training. Yep.
[35:19] >> So my perspective will be a little bit
[35:20] different because we don't typically
[35:22] build and serve models. We provide the
[35:25] the kind of data insights infrastructure
[35:27] that leads to the development of those
[35:29] models. So my perspective would be given
[35:31] the massive amounts of money being
[35:32] thrown around on compute um don't
[35:35] underinvest in the relatively small
[35:38] chunk of this cycle that's where the
[35:41] design and the deliberate development is
[35:43] actually happening. So do invest a
[35:46] reasonable amount of the budget in
[35:48] getting a a good set of goals, a
[35:51] carefully designed initiative, a data
[35:54] strategy underway and build for
[35:57] iteration to prevent a bunch of spend
[35:59] running away in the wrong direction
[36:00] later when we realized that the
[36:03] requirements or maybe the intent of this
[36:04] research cycle wasn't well designed. So
[36:07] we sometimes see companies they get
[36:08] excited about getting to the end but if
[36:10] you underinvest in that initial
[36:12] deliberate design period then you waste
[36:15] a lot later and it can be iterative. It
[36:18] can be it can be prudent and it can be
[36:19] relatively inexpensive by comparison to
[36:22] your GPUs but it's well worth it.
[36:26] >> The answer is uh much more
[36:27] straightforward for us. We uh we raised
[36:30] 17.5 million for our seed round to build
[36:33] the next generation of human data and uh
[36:38] the overwhelming majority of that is
[36:40] going towards figuring out human machine
[36:42] interactions and building models. If you
[36:44] like building stuff, come talk to me.
[36:47] Thank you for all your perspectives. So
[36:50] the key takeaway for me from this
[36:52] discussion so so far was that garbage in
[36:54] garbage out still holds true and uh um
[36:58] and it is important to always stress
[37:01] test whatever outputs we are seeing from
[37:03] different stages of the model. So we
[37:06] have last five or eight minutes left and
[37:09] I would like to open it up for the
[37:11] audience. If you have any questions
[37:13] please come forward ask your questions
[37:15] and the panelists would be happy to
[37:17] answer them for you. I think your
[37:19] question is like how do we evaluate the
[37:21] performance of our model offline versus
[37:23] online, right? So I think in that case
[37:26] obviously you might not see the
[37:28] performance as good as you were seeing
[37:30] it offline because obviously your
[37:31] offline data is saturated right you only
[37:34] have fixed amount of data that you have
[37:35] used for your training and then you have
[37:37] fixed amount of data that you're
[37:38] evaluating your model on but when your
[37:40] model goes for online serving it sees
[37:42] some of the factor which it might have
[37:44] not even seen before. So in that case
[37:46] obviously your model is not going to
[37:47] perform as good. So that's where the
[37:49] feedback in comes in. So when every time
[37:51] we have this new data coming up we need
[37:54] to make sure that we retrain our model
[37:55] with this new kind of data so that our
[37:57] distribution is still up to date from
[37:59] for what our current data is that model
[38:01] is seeing during the serving time. So I
[38:04] think it's u probably back and forth
[38:06] process. So you always have to update
[38:08] your training data so that your training
[38:09] data is up to date with whatever your
[38:11] model is going to be seeing while it
[38:12] serves the um customers
[38:15] >> right yes it's understood that sure you
[38:18] go into production and you'll see
[38:19] different data so yeah you should have a
[38:21] feedback loop from production into
[38:23] training get that but like what is the
[38:27] quantitative way to measure that in the
[38:30] case of especially rich data like vision
[38:33] like one could say yeah you can look at
[38:35] resolution for example, right? Is there
[38:38] a variation in the resolution of the
[38:40] data? But that's trivial. But I'm just
[38:42] asking if you've encountered other such
[38:45] factors which will help you measure um
[38:49] you know akin to the statistical
[38:51] measures that's easily available in
[38:54] tabular data.
[38:56] I think I can I can take on that. Um so
[38:59] definitely there are certain metrics
[39:01] like performance, accuracy, precision
[39:03] that we are keep monitoring that with
[39:04] the test data and the one model that is
[39:06] working in the production and that kind
[39:08] of gives us an idea on how the model
[39:10] performance is compared to the test run.
[39:12] But when it comes for you to actually
[39:15] tell the whe whether the like if the
[39:18] model performance is bad what's the
[39:19] reason for that the the one way to do
[39:21] that is like an AV testing method is
[39:24] just you get some data that is going in
[39:26] the production download that and you
[39:27] also have certain evaluators to see like
[39:29] how difference or how similar how
[39:31] similarity exists between the data that
[39:34] is in the online and in the existing one
[39:36] and what kind of um and usually for the
[39:39] computer vision part the data is like
[39:41] very highdimensional And for that we
[39:43] probably use something like an embedded
[39:45] space to try to visualize them and see
[39:47] what is the difference between these two
[39:48] data are. So that gives us a certain cue
[39:51] and idea on where to look at and what
[39:53] are the parameters that we can include
[39:55] in our training data which can close the
[39:57] gap between this. So I think that's the
[39:59] way to do that
[40:01] >> other perspective like semantic matching
[40:03] you can do the matching between whatever
[40:04] your online production since you already
[40:06] come into the embedding space. So you
[40:08] don't even need the embedding space and
[40:10] if you want to do it on the upper side
[40:12] you can do the semantic matching between
[40:13] the online data that you had new the new
[40:16] data and the whatever the training data
[40:17] you have to minimize that gap. So I
[40:19] think you can evaluate the feature drift
[40:21] between the your training data and your
[40:23] online data that um is causing that
[40:26] different changes in the model.
[40:28] >> So I just had two questions. One was um
[40:31] regarding the synthetic data. So, I know
[40:34] there's like kind of like the common
[40:36] incentive I think I heard here was like
[40:38] to know when to leverage the synthetic
[40:40] data versus the real data to kind of
[40:42] bridge those gaps there, but have you
[40:44] ever came across a case where the only
[40:47] thing you had to train on was synthetic
[40:49] data? Like was what strategies would you
[40:53] employ in order to make it as realistic
[40:55] as possible in order to train the most
[40:59] accurate model that you can? Then the
[41:01] second question I had was how difficult
[41:04] do you you find the shift from I guess
[41:07] going from more traditional CV methods
[41:10] to um CV methods that are more dependent
[41:13] on like machine learning and LLMs and
[41:16] things like that.
[41:21] I think that problem where you just have
[41:23] synthetic data most of the times and
[41:25] maybe like a very little to no data in
[41:27] the re real scenario is usually
[41:29] happening like more often than not. And
[41:32] I think you can think of do uh using
[41:36] synthetic data is not just to like close
[41:38] the gap between the synthetic and to
[41:40] like increase its performance in the
[41:42] real world but more like let's say you
[41:44] have an open source model which is
[41:46] really good at doing a certain task and
[41:48] then but it's trained on a certain
[41:50] different data. So you have a synthetic
[41:52] data so you can use that data to
[41:54] basically bring that model into your own
[41:56] domain. So that's the initial first type
[41:58] of domain gap that you're going to try
[42:00] to you know fulfill it through that and
[42:02] then another way is to again as we
[42:04] discussed like computer vision is a very
[42:06] high dimensional data. So you then want
[42:08] to see like the real world data and the
[42:11] synthetic data what's the dimensionality
[42:13] of that how do they represent visualize
[42:15] them in the embedding space and see the
[42:18] differences in that. If you try to see
[42:20] the differences and find the metrics
[42:21] which are kind of be able to modify them
[42:24] and close them probably probably by
[42:27] human annotators we can try that
[42:29] >> follow up on that and I think maybe the
[42:32] unfortunate answer for now is lower your
[42:34] expectations if all you have is
[42:36] synthetic data. So if you're as um as my
[42:38] colleague said if we if you're building
[42:40] on top of an open or you know an
[42:42] existing pre-trained model have a really
[42:44] really small problem statement that
[42:45] you're trying to address with that
[42:47] synthetic data. So I think we had one
[42:49] case in um similar to the strawberry
[42:51] picking pace case where we were trying
[42:53] to identify a particular types of weeds
[42:55] in a particular type of crop. I think it
[42:57] was soy. I can't remember. Um and it was
[43:00] possible to synthesize a useful data set
[43:03] to fine-tune an existing pre-trained
[43:05] vision model to detect that type of weed
[43:08] on soy. Couldn't do anything else, but
[43:10] it could do that fairly well. So having
[43:12] a really really narrow narrowly defined
[43:14] problem statement and then building on
[43:16] top of a robust pre-trained model can be
[43:19] a way to get around a data scarcity
[43:20] issue. But if you have really really
[43:22] high dimensionality even in your like
[43:24] hypothesis space for what you're trying
[43:25] to solve for it might not work very
[43:27] well.
[43:28] >> Three three watch out watch outs like I
[43:31] see all I see this um really result in
[43:34] sub-optimal results. on even if
[43:36] synthetic data let's say you manage to
[43:37] get it working for the initial part of
[43:38] the project it's it's it depends on time
[43:42] what's working today very likely will
[43:44] not work tomorrow you need to adjust and
[43:46] adapt your approach so you're const you
[43:48] constantly need to be testing to see if
[43:51] this still fits or not the second gotcha
[43:53] is when you're testing you it's not
[43:57] enough to test a model against exactly
[44:00] what it was designed to do it likely do
[44:02] well but you need to test it against
[44:04] your own use case and that includes the
[44:07] exceptions, the outliers, all of the
[44:09] stuff that are relevant to you because
[44:11] you need to understand how well this is
[44:13] doing compared to the use case you're
[44:14] actually trying to solve. And the third
[44:16] piece, which is actually a little
[44:18] ironic, we've seen synthetic a lot of
[44:19] models that generate synthetic data
[44:21] perform better after getting some some
[44:23] inputs from human data. So, it's a it's
[44:25] kind of a loop back to it, but but it's
[44:29] fine. If it works, it works, right? So,
[44:31] so just be just be open like Teresa
[44:33] said, be ready to lower your
[44:34] expectations. It's not going to be the
[44:36] magic bullet. There's no magic bullet.
[44:38] And even if it's a magic bullet today,
[44:40] tomorrow it won't be. So, just keep
[44:42] experimenting and testing and if it when
[44:44] it stops working, it stops working. If
[44:46] it works, it works.
[44:47] >> Thank you, Ahmed. Thank you so much all
[44:48] the panelists and thanks to the audience
[44:50] for asking such interesting questions.
