Apple, 7-Eleven Keep Classical Vision as Edge AI Costs Mount

Optimizing Computer Vision: Strategies, Tools, and Platforms — Ai4

Ai4

High edge inference costs and strict latency demands force production engineers across consumer tech, retail, and pharma to anchor real-time systems on classical heuristics and tiered cascading architectures rather than pure foundation models.

AI-assisted summary of Optimizing Computer Vision: Strategies, Tools, and Platforms — original source. Check the original for full context.

Source published: 2025-08-27

Detailed notes

## Background Published by Ai4 on YouTube on August 27, 2025, this panel discussion explores best practices, tools, and platforms for building and deploying production-grade computer vision systems. Moderated by Biswa Sengupta of JPMorgan Chase & Co. and featuring engineering leaders from Apple, 7-Eleven, and Optum Rx, the session examines how enterprises optimize latency, manage computational costs, and scale visual intelligence across consumer hardware, retail, and regulated healthcare environments.

## TL;DR The panel traces computer vision's trajectory from hand-crafted heuristics to convolutional networks, vision transformers, and multimodal foundation models, emphasizing that classical techniques remain vital for lightweight edge pre-processing and zero-latency fallbacks. Industry practitioners detail practical production trade-offs, including Apple's dynamic layer routing across custom silicon, 7-Eleven's hybrid edge-cloud heuristics for 20,000-SKU store catalogs, and Optum Rx's high-speed pharmaceutical pipelines optimized via quantization, distillation, and cascaded architectures. The speakers demonstrate how multimodal models and zero-shot multi-agent systems are unifying enterprise workflows from shelf auditing to document verification. Concluding with hiring and career guidance, the panel stresses prioritizing fundamental mathematics, robust system architecture, and designing for humans first, compliance second, and machines third.

## Atmosphere & style The panel is brisk, pragmatic, and refreshingly grounded in production realities rather than speculative AI hype. Biswa Sengupta moderates with an engaging, structured rhythm, prompting candid architectural comparisons and war stories across widely differing operational constraints. The dynamic between the speakers is collaborative and collegial, creating the feel of an authentic engineering post-mortem and strategy sync where latency budgets, silicon limits, and deployment economics take center stage.

## Key takeaways - [02:14](https://www.youtube.com/watch?v=IHIccATSRUY&t=134s) The computer vision paradigm has shifted from rule-based algorithmic heuristics (e.g., Canny edge detection) and supervised CNNs to zero-shot learning and multimodal generative AI APIs, drastically lowering infrastructure costs for large product catalogs. - [04:43](https://www.youtube.com/watch?v=IHIccATSRUY&t=283s) Classical computer vision algorithms remain critical on consumer devices for autofocus, video stabilization, frame preprocessing, and as zero-latency fallbacks when dedicated neural accelerators are saturated. - [07:11](https://www.youtube.com/watch?v=IHIccATSRUY&t=431s) In regulated healthcare automation, classical vision methods provide mathematical transparency, auditability, and linear computational complexity ($O(N)$), enabling real-time execution on low-power edge microcontrollers without discrete GPUs. - [09:41](https://www.youtube.com/watch?v=IHIccATSRUY&t=581s) Apple optimizes on-device inference using the Apple Neural Engine, CoreML quantization and pruning, dynamic layer routing across CPU/GPU/ANE, and a unified memory architecture that eliminates costly memory transfers. - [12:06](https://www.youtube.com/watch?v=IHIccATSRUY&t=726s) Cost-sensitive retail environments prioritize classical CV for real-time 30 fps streams, reserving lightweight deep learning for edge-tolerated frame drops and offloading non-time-sensitive analytics to cloud batch processing. - [14:09](https://www.youtube.com/watch?v=IHIccATSRUY&t=849s) To double inference throughput from 30 to 60 fps in pharmacy packaging pipelines without upgrading hardware, engineering teams combine knowledge distillation, INT8 quantization, pruning, and cascaded model architectures. - [16:16](https://www.youtube.com/watch?v=IHIccATSRUY&t=976s) Large-scale enterprise vision applications range from mail-order pharmacy automation processing 100,000 daily prescriptions to frictionless non-barcoded produce checkout and visual media auditing in retail. - [18:25](https://www.youtube.com/watch?v=IHIccATSRUY&t=1105s) Consumer edge intelligence integrates multimodal vision for live biometrics (Face ID), semantic photo clustering, LiDAR depth mapping, and context-aware on-device screen assistants that preserve privacy by processing raw pixels locally. - [25:06](https://www.youtube.com/watch?v=IHIccATSRUY&t=1506s) Multimodal LLMs act as enterprise orchestrators by converting visual media, store video, inventory data, and marketing campaigns into structured context, enabling scene-level comprehension and zero-shot query routing. - [29:31](https://www.youtube.com/watch?v=IHIccATSRUY&t=1771s) Applying autonomous agents to financial spreadsheets presents unique challenges compared to software coding assistants like Cursor, because tabular financial data requires 100% precision and local on-device execution with zero tolerance for error. - [34:56](https://www.youtube.com/watch?v=IHIccATSRUY&t=2096s) In heavily regulated industries, ML engineers should adopt a design hierarchy of 'humans first, compliance second, machines third,' prioritizing problem-first integration over disruptive algorithmic overhauls. - [37:03](https://www.youtube.com/watch?v=IHIccATSRUY&t=2223s) Long-term career success in machine learning and robotics stems from mastering fundamental mathematical principles, systems architecture, and failure recovery under live customer traffic rather than chasing ephemeral software frameworks. - [41:39](https://www.youtube.com/watch?v=IHIccATSRUY&t=2499s) While large vision models (LVMs/VLMs) offer superior general spatial reasoning, compute constraints and edge latency requirements mean lightweight CNN baselines remain essential for specialized industrial defect inspection.

## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=IHIccATSRUY&t=4s) Introductions and Panel Overview

Biswa Sengupta, Managing Director of Generative AI (Firmwide) at JPMorgan Chase & Co., introduces the panel on optimizing computer vision systems for production environments. Joining him are three machine learning practitioners operating across diverse industries: retail, consumer electronics, and healthcare automation.

Sumedh Datar introduces himself as a Staff Machine Learning Engineer at 7-Eleven, where he has worked extensively in computer vision and now heads the enterprise's Generative AI Center of Excellence. Shirsendu Halder describes his work at Apple within the Video Imaging and Applications product-oriented research team, developing features across Photos and Camera systems, following graduate work in robotics and computer science at Carnegie Mellon University (CMU). Ameya Dhamanaskar introduces his role as an AI/ML Engineer at Optum Rx, where he builds AI systems for automated home-delivery pharmacies to improve fulfillment speed, reliability, and accuracy for patients.

- **Biswa Sengupta**: Managing Director (Generative AI), JPMorgan Chase & Co. (Moderator) - **Sumedh Datar**: Staff Machine Learning Engineer & Head of GenAI Center of Excellence, 7-Eleven - **Shirsendu Halder**: CVML Engineer, Video Imaging & Applications team, Apple; CMU robotics alumnus - **Ameya Dhamanaskar**: AI/ML Engineer, Optum Rx (Home Delivery Pharmacy Automation)

---

[02:14](https://www.youtube.com/watch?v=IHIccATSRUY&t=134s) The Evolution of Vision: From Heuristic Filters to Foundation Models

Biswa frames the historical trajectory of computer vision, tracing its evolution from classical steerable filters to convolutional neural networks (CNNs), vision transformers (ViTs), and modern multimodal foundation models spanning hundreds of parameters to billions of parameters. He asks Sumedh how these technological shifts have influenced retail analytics, loss prevention, inventory tracking, and customer experience.

Sumedh reflects on beginning his computer vision career around 2014, just as deep learning emerged into the mainstream. Early implementations relied on rule-based algorithmic heuristics—such as Canny edge detection and watershed algorithms—which depended heavily on hardcoded thresholds and strictly defined geometric assumptions. If an input deviated even slightly from these hand-crafted rules, the pipeline failed. The rise of deep convolutional networks shifted this dynamic, allowing models to learn robust semantic representations capable of distinguishing dogs, cats, and commercial products.

In retail, where 7-Eleven manages catalogs exceeding 20,000 unique stock-keeping units (SKUs), identifying products across diverse store layouts and automated checkout form factors previously demanded labor-intensive supervised model training. Today, that paradigm has evolved toward zero-shot learning and multimodal generative AI APIs, drastically lowering infrastructure costs by allowing enterprises to pay per API call rather than training and serving monolithic dedicated models from scratch.

```mermaid flowchart LR accTitle: Evolution of Computer Vision in Retail subgraph S1["Classical Vision (c. 2014)"] A["Heuristic Rules & Thresholds\n(Canny Edge, Watershed)"] end subgraph S2["Deep Learning Era"] B["Supervised CNNs\n(Trained per-item / SKU)"] end subgraph S3["Modern Paradigms"] C["Zero-Shot Learning &\nGenAI Multimodal APIs"] end A --> B --> C ```

- **Scale of catalog**: 7-Eleven manages approximately 20,000 distinct SKUs that need automated visual identification. - **Paradigm shift**: Movement from rigid thresholding $\rightarrow$ supervised custom model training $\rightarrow$ zero-shot foundation models and usage-based GenAI APIs.

---

[04:43](https://www.youtube.com/watch?v=IHIccATSRUY&t=283s) On-Device Vision and Fallback Architectures at Apple

Biswa turns to Shirsendu to discuss Apple's approach to on-device machine learning, which spans everything from next-token text prediction in Messages and iOS keyboards to real-time low-light computational photography. He asks how lightweight classical vision techniques interface with complex multimodal models on resource- and energy-constrained mobile devices.

Shirsendu explains that classical computer vision algorithms remain vital in production consumer hardware. Techniques like edge and contour detection continue to underpin core real-time camera functions, including autofocus routines and video stabilization pipelines. Beyond standard execution, classical algorithms serve as a "graceful fallback" when dedicated on-device neural accelerators—such as the Apple Neural Engine (ANE) or Tensor Processing Units (TPUs)—are fully saturated with concurrent tasks.

Furthermore, feature evolution on consumer devices is deeply tied to silicon capabilities over successive hardware generations. Early implementations of features like Portrait Mode (e.g., around 2020) were restricted to human subject segmentation, whereas subsequent hardware and silicon upgrades expanded this to complex multi-subject depth maps. In modern multi-stage AI architectures, classical algorithms frequently serve as lightweight pre-processors that normalize, crop, or extract features from raw camera sensor frames before passing them into heavier deep neural networks.

- **Primary classical CV use cases**: Real-time camera autofocus, optical/electronic video stabilization, and frame pre-processing. - **Failover safety**: Classical methods act as zero-latency fallbacks when neural hardware engines are busy or thermal budgets throttle compute. - **Generational hardware progression**: Portrait mode evolution from simple person-only isolation to generalized multi-object depth estimation.

---

[07:11](https://www.youtube.com/watch?v=IHIccATSRUY&t=431s) Classical CV in Regulated & Embedded Healthcare Environments

Biswa draws a parallel between the strict compliance demands of financial services at JPMorgan Chase and the high-stakes regulatory landscape of healthcare and pharmaceuticals, asking Ameya how Optum Rx evaluates traditional vision versus modern deep learning in pharmacy fulfillment operations.

Ameya emphasizes that classical computer vision is indispensable in pharmaceutical automation due to three key requirements: explainability, reliability, and algorithmic transparency. When auditing systems that package and verify patient medications, deterministic and mathematically transparent algorithms provide verification guarantees that black-box deep learning models cannot easily match.

Computational complexity also plays a decisive role. Classical image processing algorithms typically scale with linear time complexity ($O(N)$), contrasting sharply with the quadratic ($O(N^2)$) or exponential computational profiles characteristic of modern attention-based transformer architectures. In physical fulfillment centers, cameras are often paired directly with lightweight microcontrollers embedded on sorting lines without access to discrete GPUs or cloud connections. In these space-, cost-, and latency-constrained edge environments, classical methods deliver high-throughput, real-time performance within strict hardware boundaries.

---

[09:41](https://www.youtube.com/watch?v=IHIccATSRUY&t=581s) Cost Constraints, Dynamic Layer Routing, and Apple CoreML

Biswa raises the problem of training and serving costs, noting that very few organizations possess the capital to train massive billion-parameter multimodal foundation models from scratch. He asks Shirsendu how engineering teams at Apple navigate the trade-offs between model size, inference latency, and computational budget.

Shirsendu quips that while Apple does not operate under severe budgetary constraints during model training, cost optimization becomes critical at the point of **inference on edge devices**. Delivering smooth consumer experiences requires running complex models without draining battery life or introducing thermal throttling. Apple tackles this challenge through specialized silicon and its CoreML deployment framework:

```mermaid flowchart TD accTitle: CoreML Optimization and Execution Flow subgraph Optimization["Model Optimization Pipeline"] M["Trained Neural Model"] --> Q["Quantization & Pruning"] end subgraph Routing["CoreML Dynamic Execution Engine"] Q --> DR{"Dynamic Layer Router"} DR -->|"Matrix-Heavy Layers"| ANE["Apple Neural Engine (ANE)"] DR -->|"Parallel Compute"| GPU["On-Device GPU"] DR -->|"Sequential Control"| CPU["Main CPU"] end subgraph Memory["Hardware Architecture"] ANE & GPU & CPU <--> UMA[("Unified Memory Architecture\n(Zero-Copy Bus)")] end ```

1. **Hardware Acceleration (Apple Neural Engine)**: Custom silicon designed specifically to execute high-density matrix multiplications significantly faster and with far lower energy consumption than general-purpose CPUs or GPUs. 2. **CoreML Compression & Bridge**: Tooling that serves as an intermediary between trained models and device silicon, applying model compression techniques including weight quantization and structural pruning. 3. **Dynamic Layer Routing**: Runtime scheduling protocols that dynamically split and route individual layers of a single neural network across the CPU, GPU, or ANE based on real-time hardware availability and execution efficiency. 4. **Unified Memory Architecture (UMA)**: The CPU, GPU, and ANE share a unified memory bus, eliminating expensive memory copy operations across distinct compute boundaries and dramatically reducing edge latency.

[12:06](https://www.youtube.com/watch?v=IHIccATSRUY&t=726s) Classical Computer Vision vs. Deep Learning in Cost-Sensitive Retail

Biswa Sengupta directs the discussion to Sumedh Datar, pointing out that retail is an exceptionally cost-sensitive industry. In environments like cashierless autonomous convenience stores, deploying artificial intelligence requires a constant trade-off between the physical costs of compute and the actual return on investment (ROI).

Sumedh explains that cost constraints dictate almost every architectural decision at 7-Eleven. Equipping thousands of retail locations with high-end GPU clusters is impractical, particularly when edge deployments must operate reliably despite store-level connectivity limitations. In real-time video processing scenarios where streams must run at 30 frames per second (fps), deploying heavy deep learning or object detection models introduces latency. This lag creates friction and slows down customer transactions. As a result, the team prioritizes classical computer vision algorithms wherever possible. Deep learning is reserved only for tasks that classical techniques cannot solve.

```mermaid flowchart TD accTitle: Retail Computer Vision Processing Decision Flow A["Incoming Video Stream (30 fps)"] --> B{"Can classical CV solve it?"} B -- "Yes" --> C["Edge Processing: Classical CV (Zero lag, lowest compute)"] B -- "No" --> D{"Is real-time inference required?"} D -- "Yes (Frame loss tolerant)" --> E["Edge Inference: Lightweight DL"] D -- "No (Batch / Offline ok)" --> F["Cloud Upload: Batch DL Model"] ```

When deep neural networks are unavoidable, engineering teams assess operational sensitivity: - **Frame-Drop Tolerance**: If a real-time use case can tolerate occasionally skipping a few frames, a lightweight deep learning model can run locally at the edge. - **Offline Batch Processing**: If an analytical process is not time-sensitive, video data is queued locally and uploaded to the cloud for batch processing during off-peak windows. - **Hybrid Infrastructure**: Rather than forcing an "everything to the cloud" approach, retail vision architectures blend edge heuristics with cloud workloads to control bandwidth and infrastructure spend.

---

[14:09](https://www.youtube.com/watch?v=IHIccATSRUY&t=849s) Optimizing Real-Time Pharma Pipelines: Distillation, Quantization, and Cascades

Biswa pivots to Ameya Dhamanaskar, noting that pharmacy automation sits at a strict intersection where operations must remain cost-effective while maintaining near-perfect reliability because patient health and life-saving medications are involved. Biswa asks for specific examples where compute constraints or strict regulatory demands forced the engineering team to rethink and simplify their architectural approach.

Ameya agrees that compute and cost limitations drive creative engineering rather than simply throwing more expensive GPUs at a bottleneck. He illustrates this with an Optum Rx real-time object detection pipeline engineered to process medication packaging at 60 fps. Initial deep neural network models could only achieve 30 fps, half the required processing speed. Rather than upgrading hardware, the team applied optimization techniques across the model lifecycle:

- **Knowledge Distillation**: A high-capacity "teacher" model trains a compact "student" model, transferring learned feature representations so the smaller network achieves near-identical accuracy at double the inference rate. - **Quantization**: Lowering numerical precision (e.g., converting FP32 weights and activations to INT8) reduces memory bandwidth pressure and significantly boosts inference throughput at deployment time. - **Network Pruning**: Systematically stripping redundant parameters and inactive connections from deep neural nets to minimize compute overhead without degrading precision. - **Cascaded Model Architectures**: Structuring inference into a multi-tiered pipeline where lightweight models resolve straightforward, high-frequency cases immediately, routing only ambiguous or complex edge cases to larger, computationally intensive models.

---

[16:16](https://www.youtube.com/watch?v=IHIccATSRUY&t=976s) High-Scale Industry Applications: Mail-Order Pharmacy and Frictionless Retail

The panel explores concrete production applications across their respective industries. Ameya details the massive scale of mail-order pharmacy fulfillment at Optum Rx, where facilities process roughly 100,000 prescriptions every single day. Operating at this volume requires automated vision systems integrated directly into fulfillment lines to execute automated quality assurance and verification checks. These computer vision pipelines act as a safety net, catching anomalies or packaging discrepancies that human operators might miss during high-volume shifts and significantly accelerating end-to-end conveyor throughput.

```mermaid flowchart LR accTitle: High-Throughput Mail-Order Pharmacy Vision Verification subgraph Ingestion ["Line Automation"] A["100,000 Daily Prescriptions"] --> B["High-Speed Conveyor Lines"] end subgraph Inspection ["Quality Checks"] B --> C["Automated Vision Pipeline"] C --> D{"Visual Verification"} D -- "Pass" --> E["Direct to Mail Dispatch"] D -- "Flagged" --> F["Secondary Human Review"] end ```

Sumedh details retail applications beyond the register at 7-Eleven: - **Back-Office and Media Workflows**: Utilizing multimodal large language models (MLLMs) to ingest visual marketing media, merchandising photos, and operational video feeds to automatically flag merchandising discrepancies and optimize back-office administrative decisions. - **Frictionless Non-Barcoded Product Checkout**: While packaged goods rely on standard barcode scanners, high-velocity prepared foods and produce lack barcodes. Computer vision systems recognize these items instantly to eliminate manual cashier lookups and streamline customer checkout.

---

[18:25](https://www.youtube.com/watch?v=IHIccATSRUY&t=1105s) Computer Vision on Consumer Devices: Face ID, Memories, and LiDAR Depth

Biswa turns to Shirsendu Halder to discuss the pervasive, background computer vision running inside modern consumer hardware like the Apple iPhone. Shirsendu outlines key features where deep learning executes seamlessly beneath everyday consumer interactions:

- **Live Biometric Tracking (Face ID)**: Real-time spatial tracking and facial feature validation executed instantly every time a user unlocks their device. - **Semantic Photo Clustering & "Memories" Generation**: Algorithms generate saliency maps across user photo libraries to detect salient subjects, perform face clustering to group individuals, infer the emotional mood or aesthetic theme of the scene, and automatically curate synchronized video montages set to matching audio soundtracks. - **Creative Production Tools**: Professional video editing tools (such as Final Cut Pro and Blackmagic tools) leverage segmentation models for automated background removal and green-screen-style matting without physical backdrops. - **Hardware-Assisted Depth Mapping (LiDAR)**: Combining optical camera feeds with active LiDAR depth sensors to compute high-resolution depth maps, creating realistic synthetic lens blur and depth-of-field effects in Portrait Mode.

---

[20:30](https://www.youtube.com/watch?v=IHIccATSRUY&t=1230s) Multimodal On-Device Intelligence and Agentic Workflows

Biswa notes that computer vision is no longer isolated to pixel processing, having merged with text, audio, and speech into unified multimodal systems. Shirsendu highlights how on-device intelligence is evolving toward comprehensive, holistic screen understanding and agentic OS interactions:

- **Context-Aware Screen Assistants**: Operating systems capable of perceiving whatever is active on a user's display. For instance, when a user builds a presentation slide deck, the system can parse the on-screen text context and recommend appropriate visual assets. - **Privacy-Preserving On-Device Semantics**: Sending raw visual screen pixels to cloud servers introduces significant privacy vulnerabilities. Instead, on-device vision models translate visual scenes into compact semantic representations or text embeddings locally before communicating with downstream language models. - **Agentic Creative Pipelines**: Future multimedia workflows will allow users to command complex image adjustments (such as color grading or altering the mood of an image) via natural language, with underlying frameworks like n8n orchestrating chained machine learning agents behind the scenes.

Biswa circles back to Ameya to examine how LLM-guided workflows and multimodal AI agents are set to transform industrial operations, with Ameya observing: "It is not an AI conference if we don't mention AI agents, right?" He notes that the industry is rapidly transitioning from passive visual perception toward autonomous, action-oriented systems.

[23:32](https://www.youtube.com/watch?v=IHIccATSRUY&t=1412s) Multimodal Vision in Regulated Workflows and Retail Inventory

The conversation shifts from raw visual detection toward cognitive visual understanding, where systems do not merely recognize objects but interpret context to make faster, smarter operational decisions. In healthcare and pharmacy operations, legacy workflows typically rely on standard OCR to read the text printed on a prescription. Moving beyond OCR requires semantic comprehension of the medical document, allowing large language models to run automated sanity checks and validate prescriptions against clinical rules.

A similar transformation is unfolding in retail inventory management. Instead of relying solely on manual scanning or rudimentary object counters, the next evolution involves pointing a camera directly at a store shelf to let vision-language models instantly determine what items are overstocked, what needs restocking, and how replenishment should be executed. However, expanding these vision-LLM pipelines into document and video processing introduces strict regulatory hurdles. As visual data often captures protected health information (PHI) and personally identifiable information (PII), bridging the gap between automated multimodal ingestion and strict compliance frameworks remains a critical engineering priority.

- **Prescription Validation Evolution**: Moving from flat character recognition (OCR) to semantic parsing and LLM-driven sanity checks. - **Visual Shelf Audits**: Instant shelf-state analysis determining restock urgency directly from video and image feeds. - **Privacy & Compliance**: Engineering guardrails to sanitize PHI and PII embedded across document images and enterprise video streams.

---

[25:06](https://www.youtube.com/watch?v=IHIccATSRUY&t=1506s) LLMs as Orchestrators: From Bounding Boxes to Multimodal Scene Context

In modern retail environments, multimodality allows teams to synthesize disparate data streams across the business. Beyond storefront cameras, organizations manage large volumes of media assets, inventory metrics, marketing schedules, and campaign performance data. By converting visual media into text and structured representations, engineers can programmatically inject rich contextual data directly into model prompts, positioning the multimodal LLM as an orchestrator across the entire enterprise stack.

The technological shift over the past three years highlights how rapidly computer vision expectations have evolved. Whereas drawing a basic bounding box around an isolated region of interest was once the industry's `"wow factor,"` today's multimodal LLMs can comprehend entire complex scenes—distinguishing a vehicle in the foreground while simultaneously analyzing the tree and background elements behind it. Furthermore, the operational cost dynamics have dramatically improved: querying multimodal foundation models via pay-per-token APIs is often far more cost-effective than provisioning, hosting, scaling, and maintaining dedicated deep learning models around the clock.

```mermaid flowchart TD accTitle: Multimodal Orchestrator Flow subgraph Ingestion["Multimodal Data Sources"] A["Visual Media & Store Video"] B["Inventory & Transaction Data"] C["Marketing & Campaign Feeds"] end

subgraph Processing["Context Synthesis"] D["Media-to-Text Conversion"] E["Dynamic Prompt Assembly"] end

subgraph Intelligence["Model Execution"] F["Multimodal LLM Orchestrator"] G["Actionable Retail Insights"] end

A --> D B --> E C --> E D --> E E --> F F --> G ```

---

[27:02](https://www.youtube.com/watch?v=IHIccATSRUY&t=1622s) Multi-Agent Architectures as Intelligent Routers for Enterprise Data

Multi-agent architectures are reshaping how complex user intents are routed and resolved across enterprise backends. Historically, routing a query required curated training datasets to train traditional classifiers like Support Vector Machines (SVMs) or logistic regression models to predict specific task labels. Modern LLM-driven multi-agent frameworks act as zero-shot dynamic routers, evaluating incoming prompts and directing them to the specialized agent best suited to handle the request without requiring pre-trained classification models.

In retail, this paradigm allows businesses to unify isolated silos—transaction logs, inventory databases, financial records, and visual media—into a centralized platform. By deploying specialized agents equipped with Text-to-SQL converters and document parsers, multi-agent frameworks translate disparate data structures (such as spreadsheets, relational tables, and raw images) into natural language, synthesizing holistic operational answers for non-technical stakeholders.

- **Dynamic Zero-Shot Routing**: Eliminates the overhead of training bespoke SVM or regression classifiers for query classification. - **Cross-Domain Data Blending**: Connects point-of-sale transactions, warehouse inventory, and media feeds through unified agent interfaces. - **Format Normalization**: Converts raw SQL tables, Excel spreadsheets, and visual assets into human-readable summaries.

---

[29:31](https://www.youtube.com/watch?v=IHIccATSRUY&t=1771s) Structured Productivity Tools: The "Cursor for Spreadsheets" and High-Stakes Finance

While developer-focused agentic tools like Cursor have revolutionized software engineering, productivity workflows centered around spreadsheets remain largely underserved by AI. Unlike open-ended source code—where developers can implement logic through various architectural patterns, classes, and helper functions—spreadsheets possess rigid, two-dimensional spatial structures where analytical manipulations are strictly constrained. Building on-device multimodal agents capable of parsing cell layouts, formula dependencies, and visual table boundaries represents a major frontier, particularly because the enterprise world overwhelmingly runs on local Microsoft Excel installations rather than purely cloud-based sheets.

In financial services, developing reliable agents for Word, Excel, and PowerPoint automation represents the holy grail, yet it introduces unique engineering difficulties. While prototype demonstrations can easily ingest data and generate high-level summary charts, mastering the `"messy middle"`—performing rigorous, multi-step spreadsheet computations—remains exceptionally difficult. In highly regulated sectors like pharmaceuticals and investment banking, systems do not have `"the luxury of being 95% accurate"`; operational deployment demands absolute 100% precision, turning seemingly simple spreadsheet analysis into a demanding production challenge.

---

[32:52](https://www.youtube.com/watch?v=IHIccATSRUY&t=1972s) Coordinated Autonomous Systems and Audit-Ready Pharma Supply Chains

Integrating multiple intelligent systems within pharmaceutical supply chains requires deterministic real-time communication between hardware and vision models. At automated order fulfillment stations, robotic arms physically transport medical products while high-resolution cameras simultaneously run real-time visual quality and verification inspections. Orchestrating these workflows requires a central control plane that synchronizes robotic handoffs, executes computer vision models, and monitors station throughput without introducing network latency or processing bottlenecks.

Operating in heavily regulated pharmaceutical distribution requires systems to be `"audit ready by design."` Every action within the facility is tracked through an immutable digital chain of custody. From the precise moment a drug package enters the warehouse, through robotic sorting, automated visual validation, and customer dispatch, every visual frame, inter-system message handoff, and operational timestamp is comprehensively logged to ensure total traceability.

- **Real-Time System Interoperability**: Orchestrates synchronized physical movements between robotic pickers and computer vision inspection cameras. - **Audit-Ready Architecture**: Enforces automatic logging of all inference decisions, visual frames, and network handoffs. - **Chain of Custody Enforcement**: End-to-end provenance tracking of pharmaceuticals from warehouse intake through customer fulfillment.

[34:56](https://www.youtube.com/watch?v=IHIccATSRUY&t=2096s) Navigating ML Careers in Regulated Industries: Humans First, Machines Third

The discussion turns toward career advice for professionals navigating regulated industries, addressing both technically inclined practitioners and those pursuing non-technical paths. Ameya Dhamanaskar emphasizes the necessity of cultivating a "problem-first mindset." In heavily regulated fields, the underlying technical challenges—such as optimizing for accuracy, reliability, and inference speed—tend to remain constant over time, whereas algorithms and modeling techniques continuously evolve. Staying technically competitive requires keeping pace with these algorithmic advancements while never losing sight of the core operational problem being tackled.

```mermaid flowchart LR accTitle: Three-Tier Hierarchy for Regulated System Design subgraph Hierarchy["System Design Priority"] H1["1. Humans First Preserve existing workflows"] H2["2. Compliance Second Meet regulatory standards"] H3["3. Machines Third Select and tune models"] H1 --> H2 --> H3 end ```

On the non-technical side, clear and straightforward communication across organizational boundaries is essential. Practitioners must act as effective liaisons, translating business goals gathered from stakeholders into concrete technical requirements for engineering teams. Establishing a shared vocabulary and explicitly setting expectations upfront prevents misunderstandings down the road. Furthermore, when rolling out machine learning into environments with humans in the loop, engineers must avoid being disruptive:

> "Don't be disruptive when trying to make changes in the system. A lot of our work involves humans in the loop. So you do not want to go in on day one changing and disrupting their workflows... when we develop systems, we develop for humans first, compliance second, and machines third."

- **Problem-first philosophy**: Core performance metrics (speed, reliability, accuracy) remain stable even as architectures and model families shift. - **Cross-functional liaison**: Technical professionals must establish a shared language to bridge executive business value and developer implementation. - **Workflow preservation**: Successful enterprise adoption depends on seamless integration into existing human workflows rather than abrupt overhauls.

---

[37:03](https://www.youtube.com/watch?v=IHIccATSRUY&t=2223s) Fundamentals Over Frameworks and System Architecture in Robotics and Multimodal AI

Moderator Biswa Sengupta directs the conversation toward the resurgence of robotics, hardware-software co-design, and next-generation multimodal world models. Asked what emerging engineers must focus on to crack these advanced domains, Shirsendu Halder shares what he considers the single greatest piece of career advice he ever received: prioritize fundamental mathematical and algorithmic principles over ephemeral software frameworks. Frameworks and popular architectures change constantly—transitioning from convolutional backbones to transformers today, and inevitably to novel representations capable of ingesting images, temporal sequences, and 3D meshes simultaneously tomorrow—making first-principles comprehension indispensable.

Turning research breakthroughs into impactful products demands evaluating cutting-edge innovations through a consumer-facing lens. Halder points to the landmark paper *"Attention Is All You Need"* as a prime example of research that transformed into ubiquitous consumer products like ChatGPT once OpenAI packaged the core concepts for broad user utility. While modern trends like rapid AI-assisted scripting ("vibe coding") empower individual builders to prototype rapidly, Halder cautions that engineers must maintain a rigorous grasp of overarching system architecture. Without an architectural foundation, developers inevitably hit bottlenecks when attempting to scale services across large user bases or migrate workloads to constrained edge hardware targets.

- **Foundational mathematics**: Deep mechanical understanding outlasts shifting framework trends and library abstractions. - **Research to product conversion**: Lasting impact comes from taking foundational machine learning papers and orienting them around tangible consumer needs. - **Architectural scalability**: Rapid prototyping tools enable quick builds, but scaling infrastructure and hardware transitions require deep system architecture expertise.

---

[39:17](https://www.youtube.com/watch?v=IHIccATSRUY&t=2357s) Hiring ML Talent: Real-World Customer Impact and Failure Recovery

Biswa Sengupta shifts the topic to technical recruitment, asking Sumedh Datar what qualities he looks for when evaluating candidates for machine learning teams. Datar observes that the bar for entry-level tasks has shifted dramatically because popular algorithms and boilerplate code can now be instantly generated using conversational AI tools like ChatGPT or Perplexity. Consequently, baseline coding proficiency or mere model training is no longer an adequate differentiator. What truly distinguishes exceptional candidates is their ability to deploy a product into the wild, observe real-world failure modes under actual user interaction, and systematically iterate based on that feedback:

> "It's more about building the algorithm, building the product, putting it out in front of the customer or whoever in the wild, right? And then when it does not work, and when you take that hit, and when you go back and see what's changing and understand the way people are interacting with your product... That is where you have tremendous learning."

Sengupta strongly echoes this perspective, noting that having hired hundreds of engineers, he frequently encounters resumes loaded with impressive buzzwords that collapse during live technical evaluations. Modern enterprises hire talent to move the needle on financial outcomes—either expanding the top line or reducing bottom-line operational costs. Engineers must marry rigorous theoretical knowledge in mathematics and computer science with sharp product intuition to create deployable, value-accretive systems.

- **Commoditization of code generation**: Basic algorithmic scripting and model training can be automated by LLMs; real-world problem-solving cannot. - **Resilience through field testing**: Superior engineers distinguish themselves by diagnosing erratic edge cases, model drift, and strange predictions after systems face uncurated user traffic. - **Business value alignment**: Hiring managers seek engineers who directly drive business outcomes by improving revenues (top line) or lowering operational costs (bottom line).

---

[41:39](https://www.youtube.com/watch?v=IHIccATSRUY&t=2499s) Q&A: Moving from CNNs to Large Vision Models (LVMs) and Computational Overhead

An audience member opens the floor by asking whether industry workloads are actively abandoning convolutional neural networks (CNNs) in favor of fine-tuning large vision models (LVMs) and vision-language models (VLMs), particularly for complex labeling and spatial reasoning tasks. The attendee notes concerns that current vision models may still lack genuine spatial world modeling compared to bespoke networks, raising the question of whether an ML engineer's core role will soon reduce entirely to fine-tuning foundation vision models.

```mermaid flowchart TD accTitle: Vision Model Architecture Selection Strategy Q{"Task Domain & Environment"}

Q -->|"High Latency/Compute General Multimodal Context"| VLM["Large Vision Models (LVM/VLM) • Rich spatial & temporal reasoning • General object recognition"] Q -->|"Industrial Edge / Strict Latency Specific Defect Detection"| CNN["Convolutional Networks (CNN) • Lightweight & low overhead • Specialized inspection (e.g., cracked pipes)"]

CNN -.->|"Iterative Progression Prompt & Multimodal Tuning"| VLM ```

Shirsendu Halder responds that large multimodal models have already demonstrated superior spatial and temporal reasoning over classical networks across many benchmarks. The primary roadblock preventing complete industry-wide replacement is not representational capability, but sheer computational overhead. Deploying high-parameter vision models onto resource-constrained edge devices or latency-critical hardware remains prohibitive until hardware acceleration and compression catch up.

Sumedh Datar adds that model choice is dictated by the specific operating domain. While generalized consumer concepts (such as cars, trees, or standard retail items) are easily handled by large models out of the box, high-stakes industrial applications—such as defect inspection identifying cracked manufacturing pipes—often necessitate starting with dedicated CNN architectures. Teams can master the problem domain, data distributions, and prompt structures using lightweight, specialized baselines before determining if transitioning to a heavier foundation model provides justifiable incremental value.

- **Spatial and temporal capability**: VLMs exhibit strong general visual reasoning, but edge deployment remains bottlenecked by high computational footprints. - **Domain specificity**: Everyday consumer objects benefit from foundation models, whereas precise defect detection in niche manufacturing domains often still favors custom CNN baselines. - **Stepped migration**: Practical teams typically deploy simpler convolutional baselines first to understand domain nuances before escalating to complex multimodal models.

---

[44:09](https://www.youtube.com/watch?v=IHIccATSRUY&t=2649s) Q&A: Attention Mechanisms in Vision, Vision Transformers, and Saliency Maps

A second audience member asks whether it is feasible to apply attention mechanisms analogously to computer vision—specifically allowing a model facing low classification confidence to dynamically "zoom in" or concentrate its attention on ambiguous subregions of an image to push identification accuracy beyond 99%.

Ameya Dhamanaskar points out that this conceptual transfer from natural language processing to vision has already taken place at scale through the widespread adoption of Vision Transformers (ViTs). The industry has actively transitioned beyond traditional convolutional filters to self-attention mechanisms that calculate relationships across image patch tokens for detection and segmentation tasks.

Shirsendu Halder elaborates on the historical lineage of the idea, explaining that long before transformer architectures took over the field, computer vision researchers achieved a similar visual focus using saliency maps in conjunction with CNNs:

> "Before this whole transformer thing came in, there used to be something called saliency maps, which was exactly that but for CNNs. If you have an image and there's a subject like an animal, you could generate a frame that would just tend to the regions that had the most prominent features."

Following these answers, moderator Biswa Sengupta calls for any final inquiries before thanking attendees and acknowledging the panel members—Ameya Dhamanaskar, Shirsendu Halder, and Sumedh Datar—as the session concludes.

- **Vision Transformers (ViTs)**: Transformers natively deploy attention across visual patches, directly implementing attention mechanisms for image understanding. - **Saliency maps**: Pre-transformer techniques used gradient-based saliency maps to identify, highlight, and isolate high-confidence or ambiguous visual features. - **Panel conclusion**: The panel highlights how modern visual attention builds directly on decades of computer vision feature localization techniques.

[46:16](https://www.youtube.com/watch?v=IHIccATSRUY&t=2776s) Session Wrap-Up and Acknowledgments

The session concludes as the panelists and organizers express their gratitude to the audience and each other for the insightful discussion on computer vision strategies, tools, and real-world deployment challenges.

- Panelists included Sumedh Datar (7-Eleven), Shirsendu Halder (Apple), Biswa Sengupta (JPMorgan Chase & Co.), and Ameya Dhamanaskar (Optum Rx). - The discussion highlighted key strategies for optimizing computer vision performance, selecting the right deployment platforms, and managing computational costs in real-world application environments.

## Notable quotes > [04:43](https://www.youtube.com/watch?v=IHIccATSRUY&t=283s) graceful fallback — Shirsendu Halder

> [20:30](https://www.youtube.com/watch?v=IHIccATSRUY&t=1230s) It is not an AI conference if we don't mention AI agents, right? — Ameya Dhamanaskar

> [25:06](https://www.youtube.com/watch?v=IHIccATSRUY&t=1506s) wow factor

> [29:31](https://www.youtube.com/watch?v=IHIccATSRUY&t=1771s) messy middle

> [29:31](https://www.youtube.com/watch?v=IHIccATSRUY&t=1771s) the luxury of being 95% accurate

> [32:52](https://www.youtube.com/watch?v=IHIccATSRUY&t=1972s) audit ready by design

> [34:56](https://www.youtube.com/watch?v=IHIccATSRUY&t=2096s) Don't be disruptive when trying to make changes in the system. A lot of our work involves humans in the loop. So you do not want to go in on day one changing and disrupting their workflows... when we develop systems, we develop for humans first, compliance second, and machines third. — Ameya Dhamanaskar

> [37:03](https://www.youtube.com/watch?v=IHIccATSRUY&t=2223s) vibe coding — Shirsendu Halder

> [39:17](https://www.youtube.com/watch?v=IHIccATSRUY&t=2357s) It's more about building the algorithm, building the product, putting it out in front of the customer or whoever in the wild, right? And then when it does not work, and when you take that hit, and when you go back and see what's changing and understand the way people are interacting with your product... That is where you have tremendous learning. — Sumedh Datar

> [44:09](https://www.youtube.com/watch?v=IHIccATSRUY&t=2649s) Before this whole transformer thing came in, there used to be something called saliency maps, which was exactly that but for CNNs. If you have an image and there's a subject like an animal, you could generate a frame that would just tend to the regions that had the most prominent features. — Shirsendu Halder

## Mentioned 7-Eleven, Ai4, Ameya Dhamanaskar, Apple, Apple Neural Engine (ANE), Attention Is All You Need, Biswa Sengupta, Blackmagic, Canny edge detection, Carnegie Mellon University (CMU), ChatGPT, CoreML, Cursor, Face ID, Final Cut Pro, iOS, JPMorgan Chase & Co., LiDAR, Microsoft Excel, Microsoft PowerPoint, Microsoft Word, n8n, OpenAI, Optum Rx, Perplexity, Portrait Mode, Shirsendu Halder, Sumedh Datar, Vision Transformers (ViTs), watershed algorithm

## Video description Sumedh Datar, Staff Machine Learning Engineer, 7-Eleven Shirsendu Halder, CVML Engineer, Apple Biswa Sengupta, Managing Director (Generative AI), JPMorgan Chase & Co. Ameya Dhamanaskar, AI/ML Engineer, Optum Rx

Abstracts

This session will explore the best practices, tools, and platforms for building and deploying effective computer vision systems. Panelists will discuss the latest advancements in computer vision technologies, including deep learning frameworks, pre-trained models, and cutting-edge techniques for image and video analysis. Learn how to select the right tools and platforms to optimize performance, reduce computational costs, and scale computer vision applications across various industries. The session will also highlight key strategies for achieving high accuracy in real-world environments.

[Source](https://www.youtube.com/watch?v=IHIccATSRUY)

中文简报

边缘AI成本攀升，Apple与7-Eleven仍保留传统视觉技术高昂的边缘推理成本与严格的时延要求，迫使消费科技、零售和制药领域的生产工程师将实时系统建立在传统启发式算法与分层级联架构之上，而非纯基础模型。专家组梳理了计算机视觉从手工启发式规则到卷积网络、Vision Transformer及多模态基础模型的发展历程，强调传统技术在轻量化边缘预处理和零时延回退保障中依然不可或缺。行业从业者详细分享了生产实践中的权衡考量，包括 Apple 在定制芯片上的动态网络层路由、7-Eleven 针对 20,000 种 SKU 门店商品目录的混合边缘-云端启发式方案，以及 Optum Rx 通过量化、蒸馏和级联架构优化的高速医药流水线。演讲嘉宾展示了多模态模型与零样本多智能体系统如何统一从货架审计到文档验证的企业工作流程。最后，在招聘与职业规划方面，专家组强调应重点关注基础数学、稳健的系统架构，并坚持“以人为本、合规次之、机器第三”的设计原则。计算机视觉范式已从基于规则的算法启发式技术（例如 Canny 边缘检测）和监督式 CNN 转向零样本学习与多模态生成式 AI API，大幅降低了大型产品目录的基础设施成本。 (2:14) 传统计算机视觉算法在消费级设备上依然至关重要，用于自动对焦、视频防抖、帧预处理，并在专用神经加速器饱和时充当零时延回退保障。 (4:43) 在强监管的医疗自动化领域，传统视觉方法具备数学透明度、可审计性以及线性计算复杂度（$O(N)$），能够在无需独立 GPU 的低功耗边缘微控制器上实现实时运行。 (7:11) Apple 利用 Apple 神经网络引擎（Apple Neural Engine）、CoreML 量化与剪枝、跨 CPU/GPU/ANE 的动态网络层路由，以及消除高开销内存传输的统一内存架构来优化端侧推理。 (9:41) 成本敏感的零售场景优先采用传统计算机视觉处理实时 30 fps 视频流，将轻量级深度学习留给边缘端允许丢帧的场景，并将非时间敏感的分析任务卸载至云端进行批处理。 (12:06)

Reading summary

Deploying computer vision across consumer hardware, retail, and healthcare requires balancing advanced multimodal models with classical image processing to satisfy strict latency, compute, and regulatory constraints. According to industry practitioners from Apple, 7-Eleven, and Optum Rx, foundational algorithms and hybrid execution pipelines remain indispensable for cost control and auditability. Classical Vision as an Operational Anchor (4:43) While deep learning and foundation models handle broad semantic understanding, classical algorithms like edge detection, contouring, and watershed heuristics remain core to production systems. Shirsendu Halder of Apple notes that classical methods underpin real-time camera functions like autofocus and video stabilization, serving as a zero-latency fallback when on-device neural accelerators or thermal budgets are saturated. Classical techniques also act as pre-processors to crop and normalize frames before deeper networks execute. In heavily regulated settings, algorithmic transparency is paramount. Ameya Dhamanaskar of Optum Rx highlights that classical vision algorithms typically scale with linear time complexity ($O(N)$), contrasting with the quadratic ($O(N^2)$) scaling of attention-based transformers. These deterministic techniques allow microcontrollers embedded directly on sorting lines to achieve high throughput without discrete GPUs, while offering the auditable verification guarantees required in mail-order pharmacy automation. Engineering for Edge Constraints and Latency (9:41) Hardware and bandwidth limitations dictate architectural choices when deploying models outside centralized data centers. In retail, Sumedh Datar explains that 7-Eleven prioritizes classical computer vision to process 30-fps video locally across thousands of stores, reserving deep learning for edge tasks that can tolerate dropped frames or deferring non-time-sensitive analytics to off-peak cloud batches. Halder highlights Apple's use of CoreML to apply weight quantization and structural pruning, combined with dynamic layer routing that splits a model's layers across the CPU, GPU, and Apple Neural Engine based on real-time efficiency and memory buses. When high-throughput environments demand neural networks, model optimization bridges the gap. Facing a 60-fps packaging inspection requirement that an initial deep neural network could only run at 30 fps, Optum Rx deployed knowledge distillation alongside INT8 quantization, network pruning, and cascaded architectures. This multi-tiered setup uses lightweight models for routine cases and routes only ambiguous frames to heavier models, doubling throughput without adding expensive hardware. Multimodal Expansion and Autonomous Workflows (20:30) Enterprises are expanding beyond isolated visual inspection into multimodal systems and dynamic agents. Instead of training bespoke supervised models for over 20,000 retail SKUs, Datar reports that zero-shot learning and usage-based multimodal APIs lower infrastructure costs compared to hosting monolithic models. Large multimodal models also enable inventory tracking by assessing store shelves directly from video and orchestrate enterprise data by translating images and media schedules into structured text prompts for broader analysis. On consumer devices and regulated fulfillment lines, multimodal perception is shifting toward agentic execution. Halder outlines device interfaces that parse visual screen contexts locally to protect privacy before querying language models. Dhamanaskar notes that at high fulfillment volumes—such as Optum Rx's 100,000 daily prescriptions—robotic arms and verification cameras require real-time orchestration alongside immutable digital audit trails that record every visual frame and handoff to preserve chain-of-custody compliance. Engineering Priorities in Production Deployments (34:56) The panel emphasizes that effective applied machine learning depends on foundational engineering rather than chasing ephemeral frameworks. In regulated sectors, Dhamanaskar advocates designing for 'humans first, compliance second, and machines third,' ensuring automated tools integrate seamlessly into existing operations rather than disrupting frontline workers. Halder cautions that while AI code generators allow rapid prototyping, scaling software across massive user bases or low-power edge hardware demands a deep grasp of mathematics and system architecture. From a technical hiring perspective, Datar and moderator Biswa Sengupta observe that basic model scripting and boilerplate code have become commoditized. The primary differentiator for engineers is product resilience: deploying models into production environments, systematically diagnosing unexpected failure modes and edge cases under real user interaction, and aligning system outputs with business objectives such as revenue growth or operational savings. While large vision-language models demonstrate superior spatial reasoning, high computational overhead continues to block their wholesale adoption at the edge. The panel concludes that domain-specific tasks—such as defect inspection or extreme low-latency processing—will continue to rely on simpler convolutional baselines and classical vision, adopting foundation models incrementally as compression and hardware acceleration evolve.

精炼阅读摘要

来自摩根大通、苹果、7-Eleven 与 Optum Rx 的技术专家指出，尽管多模态基础模型显著提升了场景理解能力，但在真实生产环境中，计算机视觉系统的优化与落地仍高度依赖经典确定性算法、模型轻量化压缩及软硬件协同调度，以在严苛算力、成本、能耗与合规监管约束下实现商业回报。经典算法的持久价值与确定性保障 (7:11) 在边缘计算与强监管工业场景中，传统启发式计算机视觉算法并未被深度学习完全淘汰。Apple 的 CVML 工程师 Shirsendu Halder 指出，边缘与轮廓检测等经典技术至今仍是相机自动对焦与视频防抖的核心组件；当专用神经加速硬件（如 Apple Neural Engine）因多任务满载或发热限频时，经典算法能提供零延迟的平滑回退保障，同时也在多阶段架构中承担传感器原始图像的轻量预处理工作。在高度合规的医药健康自动化领域，Optum Rx 的 AI/ML 工程师 Ameya Dhamanaskar 强调，经典图像处理算法通常具备线性时间复杂度（O(N)），相比注意力机制 Transformer 的二次方（O(N²)）或指数级复杂度，更适合部署在没有独立 GPU 的分拣线低功耗微控制器上。此外，在药品包装与核验审计中，确定性、数学上透明的经典算法能提供黑盒深度学习模型难以比拟的可解释性与审查保证。算力约束下的模型压缩与硬件动态调度 (9:41) 在边缘设备与零售终端部署视觉模型时，能效比与推理延迟直接决定了用户体验与投资回报率。Shirsendu Halder 介绍，Apple 借助专为高密度矩阵乘法定制的神经引擎与统一内存架构（UMA），消除了跨芯片计算边界的数据拷贝开销；其 CoreML 工具链结合权重量化与结构化剪枝，并在运行时动态将网络各层调度至 CPU、GPU 或 ANE，在保障流畅体验的同时抑制发热与电池消耗。面对高吞吐流水线与成本敏感的线下零售，模型压缩与混合架构是破局关键。Ameya Dhamanaskar 分享，Optum Rx 在处理 60 fps 实时药品包装检测时，最初深度模型仅能达到 30 fps；团队通过知识蒸馏（教师-学生网络迁移）、INT8 量化、网络剪枝以及级联架构（轻量模型处理高频简单样本，复杂样本后送），在不升级硬件的前提下使推理速度翻倍。7-Eleven 的 Sumedh Datar 则指出，便利店面对超过 20,000 种库存单元（SKU）与 30 fps 实时流时，同样优先依赖传统算法，并通过边缘轻量推理与离线云端批处理结合的混合架构来遏制带宽与算力成本。多模态模型与智能体驱动的业务编排 (25:06) 计算机视觉正从单纯的目标检测向融合上下文的多模态认知理解演变。Sumedh Datar 阐述，7-Eleven 正利用多模态大语言模型（MLLM）整合店内视频、营销素材与库存数据，将图像转化为语义嵌入并动态编排进提示词，实现商品陈列比对与鲜食无条码结算；按 Token 计费的基础模型 API 调用模式，相较全天候维护专用单体模型大幅降低了企业基建开销。此外，基于 LLM 的多模态智能体框架正取代传统 SVM 分类器，充当零样本动态路由，统一打通交易日志、结构化表格与视觉媒体。然而，多模态智能体向高严苛业务拓展仍面临瓶颈。Biswa Sengupta 与专家们讨论指出，在涉及投资银行与医药配药的复杂表格处理（如 Excel 自动化）中，系统没有容忍 95% 准确率的奢侈空间，必须满足 100% 的绝对确定性；同时，摄取企业单据与视频流中的健康受保护信息（PHI）与个人身份信息（PII）时，必须构建严格的数据合规防护栏。生产落地的人本层级与架构基本功 (34:56) 在技术选型与工程实践中，与会专家一致主张将真实业务约束置于算法流行度之上。Ameya Dhamanaskar 提出了受监管环境下的系统设计三层原则：“人第一、合规第二、机器第三”。他强调，算法落地绝不能粗暴破坏既有人工工作流；在 Optum Rx 每日处理约 10 万张处方的流水线中，视觉系统与机械臂协同运作，通过对每一帧图像与推理动作建立全链路不可篡改的时间戳与日志，确保系统“在设计层面随时合规受检”。关于工程人才能力建设，Shirsendu Halder 建议从业者应立足底层数学与系统架构，而非频繁更迭的软件框架，指出只有理解硬件边界与可扩展性，才能将研究突破转化为消费级产品。Sumedh Datar 与 Biswa Sengupta 补充道，在基础代码生成已高度自动化的今天，优秀工程师的关键优势在于敢于将模型推向未受控的真实用户场景，并在系统遭遇失败、数据漂移或异常边缘用例时，具备快速诊断、迭代排错并最终撬动财务回报的核心能力。总体而言，视觉大模型与注意力机制在通用语义理解上展现了强大泛化能力，但在工业缺陷检测、低功耗边缘芯片及高精度合规场景中，算力开销与黑盒缺陷依然是主要瓶颈。当前工业界的共识并非盲目全面替换，而是构建分层级联系统——将经典确定性算法、定制卷积模型与多模态基础模型协同配合，在计算成本、审计透明度与复杂推理之间取得平衡。

Detailed notes — zh

## Background 该内容由技术大会频道 Ai4 于 2025 年 8 月 27 日发布在 YouTube 平台。根据视频简介，这场行业圆桌论坛汇聚了来自 7-Eleven、苹果（Apple）、摩根大通（JPMorgan Chase）和 Optum Rx 的资深专家，共同探讨在构建与部署计算机视觉系统时的最佳工程实践、预训练模型与深度学习框架的选择、算力成本优化策略以及在跨行业真实环境中保障高精度的关键方法。 ## TL;DR 圆桌嘉宾系统梳理了计算机视觉从传统手工算子、CNN、Transformer 到多模态大模型与具身智能的演进历程，并强调经典 CV 在端侧优雅降级、线性复杂度要求与强监管可解释性中的基石地位。来自苹果、7-Eleven 和 Optum Rx 的工程师深入拆解了端侧软硬件协同推理（如 ANE 加速与统一内存架构）、模型量化剪枝、知识蒸馏及级联分流等降本增效方案，同时分享了多模态大模型与多智能体（Multi-Agent）在零售盘点、供应链质检与跨系统数据编排中的最新落地成果。针对医疗与金融等严苛业务，专家们提出了“以人为本、合规第二、机器第三”的落地原则，并建议工程师深耕底层数学与架构基础，在真实业务反馈中实现闭环迭代。 ## Atmosphere & style 整场研讨会散发着浓厚的工业界实战气息与务实克制的工程美学。主持人的风趣串联与三位一线专家的深度剖析相得益彰，既有谈及苹果在训练算力上的充裕从容与端侧极限制热功耗博弈时的会心一笑，也有深入医疗质检与金融级“100% 精确度”合规底线时的严谨沉着。对话褪去了大模型炒作的浮华，带给观众一种在严苛软硬件约束、真实业务痛点与前沿 AI 浪潮之间寻得精准平衡的硬核技术沉浸感。 ## Key takeaways - [02:14](https://www.youtube.com/watch?v=IHIccATSRUY&t=134s) 计算机视觉技术历经可控滤波器等经典算子、CNN、Transformer 到如今的多模态大模型，模型参数量从几千激增至百亿级，研发范式也从定制化训练转向零样本学习与按 Token 付费的 API 调用。 - [04:43](https://www.youtube.com/watch?v=IHIccATSRUY&t=283s) 经典计算机视觉算法（如边缘检测与轮廓提取）未被淘汰，因其低延迟和确定性仍用于相机对焦与防抖，并能在 ANE 等专用加速器过载时充当关键的“优雅降级”保底机制。 - [07:11](https://www.youtube.com/watch?v=IHIccATSRUY&t=431s) 在医药制造等强监管场景中，经典 CV 算法因具备 $O(N)$ 线性时间复杂度、高度可解释性以及对无 GPU 边缘微控制器环境的良好适应性，依然占据核心地位。 - [09:41](https://www.youtube.com/watch?v=IHIccATSRUY&t=581s) 苹果的算力成本攻坚主要集中于端侧推理，通过专用硬件（ANE）、Core ML 压缩工具（量化与剪枝）、动态层路由及统一内存架构（Zero-Copy）实现极致能效比。 - [12:06](https://www.youtube.com/watch?v=IHIccATSRUY&t=726s) 在成本敏感的零售与工业现场，AI 系统需采用分级策略：优先采用轻量级经典 CV 算法，深度学习模型分级介入，非实时数据采用离线打散及云端批处理。 - [14:40](https://www.youtube.com/watch?v=IHIccATSRUY&t=880s) 医疗高并发场景（如每日处理 10 万处方的流水线）通过知识蒸馏、模型剪枝量化及“简单案例走小模型、复杂案例走大模型”的级联架构实现性能翻倍与高可靠性。 - [20:30](https://www.youtube.com/watch?v=IHIccATSRUY&t=1230s) 端侧多模态系统强调隐私保护，最佳实践是在本地设备将原始像素转换为语义理解脱敏后再传给 LLM 编排，避免直接上传原始图像数据。 - [23:32](https://www.youtube.com/watch?v=IHIccATSRUY&t=1412s) 计算机视觉正在从单纯的目标检测向融合 LLM 的认知与智能体编排演进；LLM 充当跨图像、库存与交易日志的调度路由器，但处理 PHI/PII 隐私合规仍是关键挑战。 - [29:31](https://www.youtube.com/watch?v=IHIccATSRUY&t=1771s) 电子表格（如本地 Excel）具备高结构化与严格修改规则，是端侧多模态智能体潜力巨大的蓝海，但在金融等受监管行业中面临着必须做到 100% 精确的严苛挑战。 - [34:56](https://www.youtube.com/watch?v=IHIccATSRUY&t=2096s) 受监管行业的 AI 系统开发必须遵循“以人为先、合规第二、机器第三”原则，避免颠覆一线人员的工作流，并实现“设计即审计”的全链路日志与存证机制。 - [37:03](https://www.youtube.com/watch?v=IHIccATSRUY&t=2223s) 面对具身智能与世界模型浪潮，研究者与工程师应重数学与算法底层原理、轻上层易变框架，同时培养端到端系统架构能力与在真实业务中闭环迭代排错的能力。 - [41:39](https://www.youtube.com/watch?v=IHIccATSRUY&t=2499s) 通用视觉任务可直接利用多模态视觉模型（LVM/VLM）的零样本泛化能力，而容错率极低的工业精密质检仍需以高精度标注的 CNN 模型为起点，逐步过渡至混合与微调架构。 ## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=IHIccATSRUY&t=4s) 嘉宾介绍与圆桌开场主持人 Biswa Sengupta（摩根大通生成式 AI 董事总经理）向现场观众致谢并宣布本次讨论的主题——“优化计算机视觉：策略、工具与平台”。他幽默而谦逊地介绍自己目前负责摩根大通全公司范围内的生成式 AI 落地工作，并向大家引荐了同台的三位工业界一线工程专家，请他们逐一分享如何在日常工作中将计算机视觉真正推向生产环境。三位嘉宾依次进行了简要的自我介绍： - **Sumedh Datar**：7-Eleven 的 Staff 机器学习工程师。他在 7-Eleven 深耕计算机视觉领域多年，目前同时兼任 7-Eleven 内部生成式 AI 卓越中心（Generative AI Center of Excellence）的负责人。 - **Shirsendu Halder**：苹果公司（Apple）的 CVML 工程师，供职于面向产品的核心研究团队——视频图像与应用团队（Video Imaging and Applications）。iPhone 用户日常接触到的“照片”与“相机”核心视觉功能与算法大多出自该团队之手；在加入苹果之前，他曾是卡内基梅隆大学（CMU）机器人与计算机科学专业的研究生。 - **Ameya Dhamanaskar**：Optum Rx 的 AI/ML 工程师，主要负责家庭送药药房（home delivery pharmacies）中的自动化 AI 系统构建，旨在通过智能化方案提升药品分拣配送的准确性、可靠性并大幅压缩履约交付周期。 --- ### [02:14](https://www.youtube.com/watch?v=IHIccATSRUY&t=134s) 从传统算子到生成式 AI：零售业视角的演进历程 Biswa 带头发起讨论，首先梳理了计算机视觉的技术变迁史。他提到，早期的计算机视觉大量依赖于可控滤波器（steerable filters）等手工设计的算子；随后领域迈入了卷积神经网络（CNN）时代；再之后演进到基于 Transformer 的架构；而当下则全面步入多模态 Transformer 模型阶段。伴随着模型架构的演进，参数规模也从过去的几百上千个激增至如今的数十亿、上百亿。针对这一演进脉络，Biswa 向拥有丰富零售分析背景的 Sumedh 发问：在库存监控、防损保护（loss prevention）以及顾客行为分析等场景中，经典视觉是如何一路演化到当下的深度学习与大模型时代的？ Sumedh 回忆道，自己从 2014 年深度学习刚露头角时便进入了计算机视觉领域。在早期，工程师主要依靠 Canny 边缘检测或分水岭算法（Watershed）等经典算子：“你只能手工设定一些硬编码规则，一旦图像特征精确匹配这套规则它才起作用，稍有偏差就彻底失效，所有逻辑都绑定在人工阈值上。”随后，深度学习的爆发彻底打破了这种局限，模型第一次具备了精准分辨猫、狗等复杂模式的泛化能力。在 7-Eleven 这样的零售实体环境中，这种能力至关重要。7-Eleven 门店内售卖的商品品类极其庞大，涵盖近 20,000 个不同 SKU（库存保有单位）。传统规则根本无法对多达两万种商品做精细化区分。借助深度学习，团队得以在不同物理形态的自助结账台（checkout systems）及智能货架上落地商品视觉识别。Sumedh 总结道，技术的演进经历了三个阶段：最初是针对特定场景大规模训练定制模型；随后走向以零样本学习（Zero-Shot Learning）为代表的轻量化方案，力求用极小甚至无需训练的模型解决问题；而近一两年则迅速拥抱生成式 AI（GenAI），如今企业只需按 API 调用付费，单次推理成本极低，整个研发和落地范式在过去五六年中发生了翻天覆地的转变。 --- ### [04:43](https://www.youtube.com/watch?v=IHIccATSRUY&t=283s) 端侧视觉挑战与经典算法的“优雅降级” Biswa 将话题转至移动端与边缘计算领域，点名苹果公司在端侧机器学习（On-Device ML）上的深度布局。从用户在 iMessage 中输入文字时的下一个 Token 预测，到暗光环境下对照片进行的夜景增亮与图像恢复，苹果在能耗极其敏感、算力严重受限的移动硬件上部署了海量机器学习模型。他向来自苹果的 Shirsendu 提问：在端侧硬件约束下，模型是如何从早期的轻量级专用算法进化到兼顾视觉与文本的多模态端侧系统的？ Shirsendu 指出，尽管现代深度模型十分强大，但经典计算机视觉算法在苹果的软件栈中仍然占据极其重要的位置，从未被彻底淘汰。例如边缘检测与轮廓提取等传统算法，至今仍在相机自动对焦（Autofocus）和视频光学防抖（Video Stabilization）等高确定性、超低延迟场景中高频使用。更重要的是，经典方法在端侧构成了关键的“优雅降级”（Graceful Fallback）机制。在现代 iPhone 或 Android 旗舰设备上，重度神经网络计算通常交由专门的硬件加速器——如苹果神经网络引擎（ANE）或特定 TPU——来处理。然而，当这些专用加速器正在被其他高优先级系统任务完全占满时，经典 CV 方法就能作为可靠的保底方案快速介入，确保系统界面不发生卡顿或功能假死。此外，从产品功能迭代来看，终端 AI 的表现高度依赖于硬件能力的代际提升。Shirsendu 举例说，2020 年左右的 iPhone 人像模式（Portrait Mode）可能还只能精准分割人像主体，而随着芯片计算能力提升，如今的人像模式已经可以自适应处理各类复杂景深与非人主体。不仅如此，在现今复杂的端到端 AI 系统中，经典 CV 方法往往被置于前置预处理管线中，对原始图像进行快速归一化与特征初筛，再喂给后续庞大的数据驱动模型。两者并非替代关系，而是软硬件协同中的上下游互补。 --- ### [07:11](https://www.youtube.com/watch?v=IHIccATSRUY&t=431s) 强监管与工业场景下的选择：可解释性、线性复杂度与边缘算力 Biswa 将讨论焦点进一步延展至强监管行业。与摩根大通所在的金融业类似，医药与医疗保健体系同样受制于极其严格的法规与合规约束。他向来自 Optum Rx 的 Ameya 提问：在医药自动化物流与药房管理等高风险生产场景中，传统视觉与现代模型是如何取舍的？当 Optum 为患者与客户提供服务时，又该如何构建出一套让外界充分信赖的高水准视觉系统？ Ameya 坦言，与苹果和 7-Eleven 的经验一致，经典计算机视觉在医药工业自动化中依然占据着不可动摇的核心地位。他从三个维度剖析了为何传统技术在工业落地中仍是刚需： 1. **可解释性、可靠性与透明度**：在医药调剂等容错率极低的工业场景中，系统必须能够向监管机构和质检工程师清晰解释“为什么会做出这一判断”。相较于黑盒化的深层神经网络，基于几何规则和数学形态学的传统方法因果链条明确，审计透明度极高。 2. **计算时间复杂度（Time Complexity）**：绝大多数传统视觉算法具有线性时间复杂度 $O(N)$，而当下主流的基于注意力机制（Attention-based）的 Transformer 架构往往伴随二次方甚至更高的计算复杂度。在追求极致实时性、超高流水线吞吐量（High-Throughput）的药房分拣流水线上，低延迟是生死指标，这使得经典算法在很多实时判定节点上是不二之选。 3. **空间与预算的物理现实**：在绝大多数工厂车间和工业自动化产线内部，硬件配备往往仅是几台工业相机连接着边缘微控制器（Microcontrollers），现场根本没有高端 GPU 支持，也不允许连接云端算力。面对“无 GPU、无云端”的严苛硬件边界，工程师在很多时候只能依靠精细打磨的传统视觉算法来破局。 --- ### [09:41](https://www.youtube.com/watch?v=IHIccATSRUY&t=581s) 苹果的算力权衡：训练不差钱，端侧推理靠软硬件协同在梳理完技术演化与场景落地后，Biswa 抛出了困扰整个行业的成本问题：从经典算法跃升到 Transformer 与多模态大模型，算力成本呈指数级攀升；现实中极少有企业能够承担动辄数亿美元的大模型预训练开销。他借此向 Shirsendu 发问：身处行业顶尖技术阵营的苹果，在立项与研发中究竟是如何平衡成本约束与超大模型训练之间的矛盾的？ Shirsendu 的回答引来了现场的一阵轻快笑声：“我认为成本限制通常主要是在推理（Inference）阶段才会被严肃考虑。我的意思是，在模型训练上，我不觉得苹果有任何预算限制（"I don't think Apple has any constraint of budget"）。”但他话锋一转，严肃强调：真正的技术攻坚和成本控制硬仗，完全发生在**端侧推理**这一侧——如何让高复杂度的前沿模型在一部小小的手机上流畅、低热、低电耗地跑起来，才是苹果工程团队的核心议题。 Shirsendu 详细拆解了苹果在实现高效端侧推理时的软硬件全栈协同策略： ```mermaid flowchart TD accTitle: Apple On-Device Inference Pipeline subgraph HW ["硬件基础设施 (统一内存架构 Unified Memory)"] ANE["Apple Neural Engine (ANE) 专攻高吞吐矩阵计算"] GPU["GPU 图形与通用并行计算"] CPU["CPU 通用逻辑调度与计算"] end subgraph SW ["Core ML 框架中间层"] Opt["模型压缩技术 (量化 Quantization / 剪枝 Pruning)"] Router["动态路由协议 (Dynamic Routing) 逐层自适应分发任务"] end Model["端侧视觉/多模态模型"] --> Opt Opt --> Router Router -->|"低功耗/密集矩阵计算"| ANE Router -->|"图形密集型计算"| GPU Router -->|"通用控制/保底计算"| CPU ``` - **专用加速硬件（ANE）**：Apple Neural Engine 专为矩阵运算定制设计，完成同等规模的矩阵乘加计算不仅耗时远低于 CPU 或 GPU，而且单位能效比极高。 - **Core ML 桥接与模型压缩**：Core ML 充当了上层算法模型与苹果底层硬件生态之间的核心桥梁。它内置了高度成熟的压缩工具链，涵盖权重量化（Quantization）与网络剪枝（Pruning）等技术，在最大化保留精度的同时大幅缩减模型体积。 - **自适应动态路由（Dynamic Layer Routing）**：Core ML 配备了动态调度协议，能够将大型模型的不同神经网络层根据硬件负载自适应地拆分并路由到 CPU、GPU 或 ANE 上混合执行，实现全设备算力的削峰填谷。 - **统一内存架构（Unified Memory）**：苹果自研芯片的 CPU、GPU 与 ANE 共享统一的物理内存总线。这意味着模型权重和图像张量在不同计算核心之间流转时，无需进行耗时且耗电的跨总线内存拷贝（Zero-Copy），从底层彻底根治了移动端深度推理中昂贵的内存带宽开销。 ### [12:06](https://www.youtube.com/watch?v=IHIccATSRUY&t=726s) 零售场景下的成本平衡与技术架构抉择在零售这样对成本极其敏感的行业中，AI 系统的部署面临着严重的硬件与环境约束。7-Eleven 的 Sumedh Datar 指出，在门店端部署庞大的 GPU 集群是不切实际的，况且许多线下门店还会遇到网络连接不稳定等限制。在需要保证每秒 30 帧（30 FPS）实时处理的场景中，如果盲目采用过于沉重的深度学习目标检测模型，极易导致帧率下降、系统卡顿与决策延时，从而给顾客带来极差的体验。因此，技术团队需要建立起非常明确的权衡机制： * **传统计算机视觉（Classical CV）优先：** 在性能受限但实时性要求高的环节，优先选用计算开销小的传统视觉算法。 * **深度学习分级套用：** 仅在传统算法无法解决问题时才引入深度学习模型；对于允许丢帧的非极其敏感场景，进行适度剪裁。 * **离线与云端批处理：** 对于时效性要求不高、允许延迟分析的数据，采用离线打散、定时分批上传至云端（Cloud Batching）的方式处理。 --- ### [14:40](https://www.youtube.com/watch?v=IHIccATSRUY&t=880s) 医疗与医药领域的性能优化实战在医药与医疗健康领域（如 Optum Rx），系统不仅要做到高成本效益，更要具备极高的可靠性，因为这直接关乎患者的用药安全与生命健康。Ameya Dhamanaskar 强调，面对计算资源有限或实时性要求极高的问题时，绝对不能简单地通过“堆叠 GPU 或硬件”来硬抗，而必须设计出更智能、更轻量化的架构方案。 ```mermaid accTitle: 医疗视觉系统性能优化与级联推理架构图 flowchart TD subgraph S1["轻量化与部署优化"] A["高精度大模型 (Teacher)"] -->|"知识蒸馏 (Knowledge Distillation)"| B["小型蒸馏模型 (Student)"] B --> C["剪枝与量化 (Pruning & Quantization)"] C --> D["极低延迟推理 (Low-precision Inference)"] end subgraph S2["级联处理架构 (Cascaded Architecture)"] E["输入图像/视频流"] --> F{"复杂度判别器"} F -->|"简单/常规案例"| G["轻量模型 (快速响应)"] F -->|"复杂/边缘案例"| H["大参数高精度模型"] end ``` 以真实项目为例，Optum Rx 团队曾开发过一个要求达到 60 FPS 的实时目标检测流水线，但初始模型仅能达到 30 FPS。他们通过以下技术路线实现了性能翻倍： * **知识蒸馏（Knowledge Distillation）：** 利用大模型的知识指导小模型训练，使小模型在保持高精度的同时显著提升运行速度。 * **量化与剪枝（Quantization & Pruning）：** 在部署阶段使用低精度推理降低计算开销，并剔除深层神经网络中的冗余参数。 * **级联架构（Cascades）：** 将简单常见的案例分流给小模型快速处理，仅将少数复杂边缘案例交由大模型判别，从而实现整体性能与成本的精准平衡。 - **Optum Rx 业务规模：** 药房邮件寄递部门每日需要处理约 100,000 份处方（100,000 prescriptions/day）。 - **视觉质检应用：** 在流水线上引入计算机视觉质量检测，用来捕捉人工肉眼可能遗漏的瑕疵，大幅提升端到端的交付吞吐量。 --- ### [16:15](https://www.youtube.com/watch?v=IHIccATSRUY&t=975s) 计算机视觉的核心行业应用场景各行各业根据其独特的业务痛点，衍生出了多样化的计算机视觉落地场景： | 行业领域 | 核心应用场景与解决方案 | 解决的具体业务问题 | |:--- |:--- |:--- | | **医药物流 (Optum Rx)** | 自动化处方质检系统 | 在每日 10 万级的流水线上进行自动视觉复核，防止人工漏检，提升出货效率与安全性。 | | **零售自动化 (7-Eleven)** | 无人/无感收银、媒体与商品识别 | 解决部分高销量商品无法扫描条形码的痛点，消除结算摩擦；结合多模态大模型（Multimodal LLMs）分析后台营销与商品图片。 | | **消费电子 (Apple)** | Face ID 实时人脸追踪、照片墙自动剪辑（Memories）、LiDAR 深度感知 | 实现无感安全解锁；通过显著性图（Saliency Maps）识别人物并自动聚类聚合成音乐视频；配合 LiDAR 生成人像模式虚化深度图。 | --- ### [20:30](https://www.youtube.com/watch?v=IHIccATSRUY&t=1230s) 端侧多模态与 AI Agent 架构展望 Apple 的 Shirsendu Halder 剖析了计算机视觉从单纯的图像分析向混合模态（Mixed Modality）及端侧 AI 演进的趋势。未来设备将更加注重全景式的交互理解，例如系统可以直接感知用户屏幕上的操作上下文（如制作 PPT），并智能推荐匹配的配图或主题。针对消费级设备，**隐私保护（Privacy）** 是至关重要的基石。直接向云端发送原始像素数据（Raw Pixel Data）会带来极大的隐私风险。因此，未来的最佳实践是在本地设备上将视觉数据转化为语义理解（Semantic Understanding），再将脱敏后的语义信息提供给大语言模型进行描述生成或任务编排。结合 N8N 等自动化工作流工具，未来的视觉创作系统有望让用户仅凭口头指令，即可完成图像色调调整、绿幕抠图乃至整体意境的重构。Optum Rx 的 Ameya Dhamanaskar 也指出，行业正全面从单一的视觉感知（Perception）向结合大语言模型的 AI Agent 工作流演进。 ### [23:32](https://www.youtube.com/watch?v=IHIccATSRUY&t=1412s) 从被动感知到认知理解：计算机视觉与大语言模型的融合演进 Ameya Dhamanaskar 指出，当前的计算机视觉技术正在发生深刻转变，从单纯的图像识别向能够理解上下文的“认知”阶段演进，核心目标是推动系统做出更快、更明智的决策。在医疗与制药工作流中，处方审核（Prescription Validation）是典型应用之一。传统的 OCR 方案仅能识别图像中的文字，而结合大语言模型（LLM）后，系统能够实现语义层面的深度理解，并通过 LLM 进行合理性与一致性校验（Sanity Checks），从而极大提升处方验证系统的准确度与业务价值。类似的技术跨越同样出现在库存管理领域。过去需要繁琐设定的流程，未来将转变为直接将摄像头对准货架，由系统自主分析哪些商品需要补货、哪些商品库存过剩以及应如何科学补货。此外，针对文档处理与日常办公，虽然公众已习惯使用 LLM 进行邮件修改或文本润色，但当工作流延伸至包含图像和视频的多模态输入时，如何处理其中包含的受保护健康信息（PHI）与个人身份信息（PII），使 LLM 严格遵守合规性与监管条例，成为了当前必须攻克的关键技术差距。 ```mermaid flowchart TD accTitle: Multi-modal Vision and LLM Evolution subgraph Traditional["传统感知模式"] A["传统 OCR / 目标检测"] --> B["纯像素级文字提取 / 边界框定位"] end subgraph Modern["多模态认知模式"] C["多模态输入 (图像 / 货架视频 / 处方)"] --> D["多模态大模型 (LLM/VLM)"] D --> E["上下文语义理解 & 合规校验"] D --> F["智能决策 (补货调度 / 处方合规审计)"] end ``` ### [25:06](https://www.youtube.com/watch?v=IHIccATSRUY&t=1506s) 零售场景下的多模态编排与低成本架构 Sumedh Datar 探讨了多模态模型在零售与门店运营中的最新落地实践。除了传统的店面货架盘点，多模态最强大的应用在于跨异构数据源的综合编排。企业内部沉淀了海量的媒体信息、库存数据、营销活动（Campaign）以及交易日志，LLM 在其中充当了智能编排器（Orchestrator）的角色，开发者可以通过代码将多源异构数据注入提示词中，一站式获取业务洞察。 Sumedh 强调，回顾短短三年前，计算机视觉领域的“高光时刻”还停留在为感兴趣区域（ROI）打上目标边界框（Bounding Box）并识别出“前方有一辆车”；而如今的多模态大模型不仅能识别前景车辆，还能自然理解背景中的树木及其相互关系。更重要的是架构与成本层面的革命：相比于独立部署、长期托管并维护庞大的传统专用深度学习模型，直接调用多模态 API 按 Token 付费的模式大幅削减了基础设施运营成本。 - **编排器角色**：LLM 打通了文本、图像媒体、库存以及营销活动的壁垒，免去了针对单一任务独立开发模型的繁琐流程。 - **降本增效**：从“常驻算力服务器托管模式”演化为“按 API / Token 付费调用模式”，显著降低了维护负担。 ### [27:02](https://www.youtube.com/watch?v=IHIccATSRUY&t=1622s) 多智能体路由机制与端到端结构化数据落地谈及当前备受瞩目的多智能体（Multi-Agent）系统，Sumedh 解释了其与传统机器学习架构的根本差异。以往为了根据用户提问将其分流至特定领域模块，团队需要准备充足的标注数据来训练逻辑回归（Logistic Regression）或支持向量机（SVM）分类器；而在多智能体体系中，LLM 扮演了极其出色的“智能路由器”（Router），即使在没有任何微调数据的情况下，仅凭对提问意图的理解，就能相对精准地将任务分派给下游专属 Agent。在零售行业端到端产品的落地方面，Sumedh 指出，企业的数据形态极为分散，涵盖交易记录、库存信息、财务报表及监控图像。通过结合 Text-to-SQL 等转换工具，多智能体系统能够将 Excel 表格、图像或关系型数据库中的复杂数据自动翻译为自然语言，再由核心 LLM 进行逻辑归纳与交叉核验，为决策层输出具备极高可解释性的业务结论。 ### [29:31](https://www.youtube.com/watch?v=IHIccATSRUY&t=1771s) 端侧生产力重塑：电子表格智能体与金融场景的严苛要求 Shirsendu Halder 提出了一个被广泛忽视却潜力巨大的领域——电子表格（Spreadsheets）智能体。当前像 Cursor 这样的 AI 编程助手已经在代码生成领域全面开花，但在全球商业运转基石的电子表格（尤其是广泛部署在端侧本地的 Excel）上，尚缺乏成熟的智能体方案。与编写代码时可以自由定义类、函数或拆分文件不同，表格中的数据具备高度结构化且修改规则极其客观、受限，这极度依赖强大的端侧多模态理解能力。 Biswa Sengupta 对此深表赞同，并指出在金融服务业中，如果能彻底打通 Word、Excel 和 PowerPoint 的多模态智能流转——从阅读与生成 Word 文档、深度分析与构建 Excel 报表，再到自动生成汇报 PPT，就能直接解决金融行业绝大多数的痛点。然而，数据摄入与最终图表展示之间的“混乱中间层”（Messy Middle Work）极难处理。Biswa 特别强调，金融与医药等强监管行业与一般互联网产品有着本质区别： > “在受监管的行业中，你没有‘95% 准确率’这种容错空间，你必须做到 100% 精确，否则系统根本无法上线。” 看似在 Demo 或 MVP 阶段表现良好的 Excel 分析系统，在走向生产环境时面临着极其严苛的工程与合规挑战。 ```mermaid flowchart LR accTitle: Financial Workflow and the Messy Middle A["非结构化/结构化数据摄入 (Word / Excel / 报表)"] --> B["核心瓶颈：'混乱中间层' (复杂多表计算 & 100% 精度校验)"] B --> C["自动化生成 (PowerPoint / 终版财务分析)"] ``` ### [32:55](https://www.youtube.com/watch?v=IHIccATSRUY&t=1975s) 制药供应链中的多智能体协同与可追溯性闭环 Ameya Dhamanaskar 深入分享了 Optum Rx 在药品供应链中调度多智能体系统的实际工程经验。在包含自动化机器人搬运、多视角工业相机视觉质检以及实时性能分析的高并发履约工作站中，最大的挑战在于如何实现多个智能系统之间的低延迟实时通信与任务交接（Handoff）。为了确保医疗供应链的绝对安全，整个多系统架构在设计之初就贯彻了“设计即审计”（Audit Ready by Design）的原则： - **全链路追踪**：从药品进入仓储库房、经由机械臂与质检设备处理，到最终打包分发给客户，完整的“监管链”（Chain of Custody）贯穿始终。 - **全要素日志存证**：系统间每一次通信握手、工步交接、质检拍摄的图像以及高精度时间戳均被完整持久化记录，确保全流程可回溯、可审计。 ### [34:56](https://www.youtube.com/watch?v=IHIccATSRUY&t=2096s) 受监管行业中的从业建议：以人为本与跨领域沟通在探讨如何在医疗、金融等受强监管行业开展 AI 职业生涯时，嘉宾们分别从技术和非技术维度分享了关键建议。首先，在技术路线上必须具备“问题优先”（problem-first）的思维模式：在实际业务中，系统对准确率、可靠性和运行速度等核心问题的追求是恒定不变的，而底层的算法和框架则在不断迭代更新，因此持续跟踪最新算法是工程师的基本功。而在非技术与协作维度，清晰简单的跨团队沟通能力至关重要。从业者往往需要充当业务利益相关者（Stakeholders）与底层开发团队之间的桥梁，将抽象的商业价值诉求精准翻译为可落地的技术需求。为了避免沟通断层，团队内部必须建立一套通用的业务-技术语言，并在项目初期设定合理的预期目标。更为重要的是在系统落地过程中切忌“颠覆式破坏”，由于医疗等领域深度依赖人机协同（Human-in-the-loop），新系统绝不能在上线第一天就强行打乱一线人员现有的操作工作流，而是应当实现无缝嵌入。 ```mermaid flowchart TD accTitle: 受监管行业系统落地优先级 subgraph Priorities ["系统开发核心优先级"] H["以人为本 (Humans First) 保障工作流无缝融合"] C["合规为重 (Compliance Second) 遵循监管与行业标准"] M["机器次之 (Machines Third) 算法迭代与算力优化"] end H --> C --> M ``` - **三大开发层级原则**：团队在构建系统时始终恪守“以人为先、合规第二、机器第三”（"develop for humans first, compliance second, and machines third"）的黄金法则。 - **推动落地的关键路径**：技术的真正影响力来自于高采纳率，尊重既有工作流并实现渐进式集成是提升系统实际使用率的前提。 --- ### [37:03](https://www.youtube.com/watch?v=IHIccATSRUY&t=2223s) 具身智能与世界模型浪潮：重基础轻框架与系统架构思维面对机器人技术复兴、软硬件协同设计（HW/SW Co-design）以及多模态世界模型（World Models）的飞速发展，与会专家对想要深耕该领域的研究者提出了底层认知上的建议。最核心的一条忠告是“专注于底层基础而非上层框架”（"focusing on the fundamentals rather than frameworks"）。深度理解底层的数学推导与算法运行原理才是立身之本，因为具体框架与模型形态每隔几年就会翻新——今天流行 Transformer，明天可能就会演进为能够同时高效处理 3D 网格（3D meshes）与图像的全新架构。此外，最具行业影响力的产品往往诞生于“将前沿学术研究与消费者视角深度结合”的交叉点，OpenAI 将《Attention Is All You Need》论文转化为风靡全球的 GPT 便是典型范例。谈及当下流行的辅助编程趋势，嘉宾指出虽然借助 AI 工具进行“氛围编码/辅助开发”（vibe coding）能让个人快速构建原型，但当系统需要面对海量并发用户或迁移至异构硬件平台时，对端到端整体系统架构（System Architecture）的宏观把控能力才是决定系统能否成功扩展的核心壁垒。 --- ### [39:17](https://www.youtube.com/watch?v=IHIccATSRUY&t=2357s) 机器学习团队的招聘标准：在真实业务中闭环迭代随着大语言模型和 Perplexity、ChatGPT 等问答工具的普及，单纯写出通用算法代码已不再具备稀缺性。在谈及团队招聘的核心考量时，工程主管们明确表示，他们更加看重候选人将模型置于真实场景（"in the wild"）并接受业务检验的实战能力。当模型首次面对真实用户并不可避免地出现异常预测或低级错误时，工程师能否承受打击、深入分析用户交互轨迹并识别失败根因，进而完成针对性的数据重训与迭代优化，是衡量其实战能力的关键指标。在实际招聘场景中，许多求职者的简历上堆砌了各种热门关键词，但一到深度技术面试环节便破绽百出。企业真正渴望的是既精通底层数学原理与计算机科学基础，又具备敏锐产品意识的复合型人才。毕竟在商业组织中，技术团队的终极使命是赋能业务——通过降本增效直接为公司的顶线增长（Top line）或底线利润（Bottom line）创造实质性价值。 --- ### [41:39](https://www.youtube.com/watch?v=IHIccATSRUY&t=2499s) 现场问答：大型视觉模型（LVM/VLM）与传统 CNN 的应用边界在观众提问环节，有参会者询问团队是否正在将工作负载从传统的 CNN 架构与标注训练迁移到大型视觉模型（LVM/VLM）上，以及当前 LVM 是否因缺乏空间理解等“世界模型”特性而存在落地瓶颈。 ```mermaid flowchart LR accTitle: 视觉模型选型路径决策 Task["视觉任务分类"] Task --> General["日常通用物体 (树木、车辆、消费品等)"] Task --> Industrial["严苛工业/制造业质检 (特定管道缺陷、内部裂纹)"] General --> VLM["大型多模态视觉模型 (LVM/VLM) 零样本泛化 / Prompt 调优"] Industrial --> CNN["传统 CNN / 定制小模型 精准标注训练"] CNN -.->|"逐步摸清数据与提示工程"| Hybrid["混合架构 / 微调 LVM"] ``` 针对这一疑问，专家指出多模态大模型在宏观的空间与时序理解能力上实际上展现出了极强优势，目前阻碍其完全取代传统小模型的首要物理瓶颈在于极其高昂的计算开销（Computational overhead）。当算力与端侧部署成本得到进一步优化后，视觉大模型的覆盖面将大幅拓宽。但在具体的业务落地决策上，需高度依赖应用场景： - **通用日常视觉任务**：对于树木、车辆、标准消费品等常见目标的识别与语义理解，LVM/VLM 展现出了卓越的泛化与跨模态交互能力。 - **制造业等垂直工业质检**：在识别特定破损管道与合格管道等高精度、容错率极低的工业场景下，最稳妥的工程路径仍然是从高精度标注的 CNN 模型切入；在摸透数据特征并跑通提示工程与评估体系后，再逐步向大型视觉模型过渡。 --- ### [44:09](https://www.youtube.com/watch?v=IHIccATSRUY&t=2649s) 现场问答：注意力机制在计算机视觉中的演进与历史溯源现场观众进一步提问：大语言模型中广泛使用的注意力机制（Attention Mechanism）是否能迁移至计算机视觉领域，以针对图像中置信度较低的局部区域进行自适应聚焦，进而突破 99% 的识别准确率瓶颈。专家们回应指出，这种跨模态的技术迁移实际上早已发生并成为行业主流——视觉 Transformer（Vision Transformers, ViT）正是将自注意力机制全面引入视觉特征提取的代表，彻底改变了以往单纯依赖 CNN 局部卷积核的范式。从计算机视觉的发展史来看，在 Transformer 席卷视觉领域之前，学术界和工业界就已广泛使用“显著性图”（Saliency Maps）技术来达成类似目标。显著性图能够在 CNN 的特征层生成针对主体（如特定动物或目标对象）的热力响应区域，自动将模型的表征与算力重点倾斜至图像中最关键的像素区域，两者在解决局部聚焦与置信度增强的核心逻辑上可谓一脉相承。随后，主持人对全场嘉宾与观众致谢，研讨环节在热烈掌声中圆满收官。 ### [46:16](https://www.youtube.com/watch?v=IHIccATSRUY&t=2776s) 研讨会总结与致谢座谈会（Panel Discussion）在热烈的掌声与嘉宾们的相互致谢中圆满结束。主持人与各位来自行业顶尖企业（包括 7-Eleven、Apple、JPMorgan Chase & Co. 以及 Optum Rx）的计算机视觉与机器学习专家共同总结了构建、部署与优化计算机视觉系统（Computer Vision Systems）的核心策略。与会嘉宾表达了对彼此分享实战经验与前沿见解的感谢，同时也向现场观众及线上参与者致以谢意，感谢大家的专注聆听与积极参与。 - 本座谈会重点探讨了计算机视觉在工业界落地时的模型性能优化、算力成本控制、跨平台扩展以及在真实复杂环境下的高精度保持策略。 - 嘉宾们共同强调了结合预训练模型（Pre-trained Models）、深度学习框架（Deep Learning Frameworks）与最新生成式 AI / 计算机视觉技术在实际业务场景中的重要价值。 ## Notable quotes > [02:14](https://www.youtube.com/watch?v=IHIccATSRUY&t=134s) 你只能手工设定一些硬编码规则，一旦图像特征精确匹配这套规则它才起作用，稍有偏差就彻底失效，所有逻辑都绑定在人工阈值上。 — Sumedh Datar > [07:11](https://www.youtube.com/watch?v=IHIccATSRUY&t=431s) 绝大多数传统视觉算法具有线性时间复杂度 O(N)，而当下主流的基于注意力机制的 Transformer 架构往往伴随二次方甚至更高的计算复杂度。 — Ameya Dhamanaskar > [09:41](https://www.youtube.com/watch?v=IHIccATSRUY&t=581s) 我认为成本限制通常主要是在推理（Inference）阶段才会被严肃考虑。我的意思是，在模型训练上，我不觉得苹果有任何预算限制（"I don't think Apple has any constraint of budget"）。 — Shirsendu Halder > [14:40](https://www.youtube.com/watch?v=IHIccATSRUY&t=880s) 面对计算资源有限或实时性要求极高的问题时，绝对不能简单地通过“堆叠 GPU 或硬件”来硬抗，而必须设计出更智能、更轻量化的架构方案。 — Ameya Dhamanaskar > [20:30](https://www.youtube.com/watch?v=IHIccATSRUY&t=1230s) 直接向云端发送原始像素数据（Raw Pixel Data）会带来极大的隐私风险。 — Shirsendu Halder > [25:06](https://www.youtube.com/watch?v=IHIccATSRUY&t=1506s) 相比于独立部署、长期托管并维护庞大的传统专用深度学习模型，直接调用多模态 API 按 Token 付费的模式大幅削减了基础设施运营成本。 — Sumedh Datar > [29:31](https://www.youtube.com/watch?v=IHIccATSRUY&t=1771s) 在受监管的行业中，你没有‘95% 准确率’这种容错空间，你必须做到 100% 精确，否则系统根本无法上线。 — Biswa Sengupta > [34:56](https://www.youtube.com/watch?v=IHIccATSRUY&t=2096s) develop for humans first, compliance second, and machines third — Ameya Dhamanaskar > [37:03](https://www.youtube.com/watch?v=IHIccATSRUY&t=2223s) focusing on the fundamentals rather than frameworks — Shirsendu Halder > [41:39](https://www.youtube.com/watch?v=IHIccATSRUY&t=2499s) 在识别特定破损管道与合格管道等高精度、容错率极低的工业场景下，最稳妥的工程路径仍然是从高精度标注的 CNN 模型切入；在摸透数据特征并跑通提示工程与评估体系后，再逐步向大型视觉模型过渡。 — Panelists ## Mentioned Biswa Sengupta, Sumedh Datar, Shirsendu Halder, Ameya Dhamanaskar, 7-Eleven, Apple, JPMorgan Chase & Co., Optum Rx, Carnegie Mellon University (CMU), iPhone, Apple Neural Engine (ANE), Core ML, Face ID, N8N, Cursor, Perplexity, ChatGPT, OpenAI, Attention Is All You Need, Microsoft Excel, Microsoft PowerPoint, Microsoft Word ## Video description Sumedh Datar, Staff Machine Learning Engineer, 7-Eleven Shirsendu Halder, CVML Engineer, Apple Biswa Sengupta, Managing Director (Generative AI), JPMorgan Chase & Co. Ameya Dhamanaskar, AI/ML Engineer, Optum Rx Abstracts This session will explore the best practices, tools, and platforms for building and deploying effective computer vision systems. Panelists will discuss the latest advancements in computer vision technologies, including deep learning frameworks, pre-trained models, and cutting-edge techniques for image and video analysis. Learn how to select the right tools and platforms to optimize performance, reduce computational costs, and scale computer vision applications across various industries. The session will also highlight key strategies for achieving high accuracy in real-world environments. [Source](https://www.youtube.com/watch?v=IHIccATSRUY)

Source quotes

[2:14] Sumedh Datar: 你只能手工设定一些硬编码规则，一旦图像特征精确匹配这套规则它才起作用，稍有偏差就彻底失效，所有逻辑都绑定在人工阈值上。 [7:11] Ameya Dhamanaskar: 绝大多数传统视觉算法具有线性时间复杂度 O(N)，而当下主流的基于注意力机制的 Transformer 架构往往伴随二次方甚至更高的计算复杂度。 [9:41] Shirsendu Halder: 我认为成本限制通常主要是在推理（Inference）阶段才会被严肃考虑。我的意思是，在模型训练上，我不觉得苹果有任何预算限制（"I don't think Apple has any constraint of budget"）。 [14:40] Ameya Dhamanaskar: 面对计算资源有限或实时性要求极高的问题时，绝对不能简单地通过“堆叠 GPU 或硬件”来硬抗，而必须设计出更智能、更轻量化的架构方案。 [20:30] Shirsendu Halder: 直接向云端发送原始像素数据（Raw Pixel Data）会带来极大的隐私风险。 [25:06] Sumedh Datar: 相比于独立部署、长期托管并维护庞大的传统专用深度学习模型，直接调用多模态 API 按 Token 付费的模式大幅削减了基础设施运营成本。 [29:31] Biswa Sengupta: 在受监管的行业中，你没有‘95% 准确率’这种容错空间，你必须做到 100% 精确，否则系统根本无法上线。 [34:56] Ameya Dhamanaskar: develop for humans first, compliance second, and machines third [37:03] Shirsendu Halder: focusing on the fundamentals rather than frameworks [41:39] Panelists: 在识别特定破损管道与合格管道等高精度、容错率极低的工业场景下，最稳妥的工程路径仍然是从高精度标注的 CNN 模型切入；在摸透数据特征并跑通提示工程与评估体系后，再逐步向大型视觉模型过渡。

Source quotes

[4:43] Shirsendu Halder: graceful fallback [20:30] Ameya Dhamanaskar: It is not an AI conference if we don't mention AI agents, right? [25:06] wow factor [29:31] messy middle [29:31] the luxury of being 95% accurate [32:52] audit ready by design [34:56] Ameya Dhamanaskar: Don't be disruptive when trying to make changes in the system. A lot of our work involves humans in the loop. So you do not want to go in on day one changing and disrupting their workflows... when we develop systems, we develop for humans first, compliance second, and machines third. [37:03] Shirsendu Halder: vibe coding [39:17] Sumedh Datar: It's more about building the algorithm, building the product, putting it out in front of the customer or whoever in the wild, right? And then when it does not work, and when you take that hit, and when you go back and see what's changing and understand the way people are interacting with your product... That is where you have tremendous learning. [44:09] Shirsendu Halder: Before this whole transformer thing came in, there used to be something called saliency maps, which was exactly that but for CNNs. If you have an image and there's a subject like an animal, you could generate a frame that would just tend to the regions that had the most prominent features.

Original transcript

[0:04] Thank you so much for coming here and um [0:08] welcome to the session on optimizing [0:10] computer vision and I'm um Bishwa Sang [0:15] Gupta. I have the pleasure of leading a [0:18] lot of generative AI for firmwide for JP [0:21] Morgan Chase. Um and I also have the [0:24] pleasure of having uh these three [0:26] gentlemen here who' be telling us about [0:29] their experiences on productionizing [0:32] computer vision in their respective um [0:35] day job. Um so just to introduce quickly [0:39] and then uh I'll ask the participants to [0:42] uh introduce themselves. We have Sumed [0:45] from 7-Eleven, [0:47] uh Shishendu from Apple and Amir from [0:51] Optum. Um so we'll just start with a [0:54] quick introduction. So Shid if you want [0:56] to go first. [0:58] Sure. [1:00] >> Hey everyone, hope you had a wonderful [1:03] lunch. So I'm Sume. I'm a staff machine [1:07] learning engineer with 7-Eleven. Um I [1:10] worked uh extensively in the computer [1:12] vision space for sun 11 and currently I [1:16] head the genai center of excellence [1:18] within 71. [1:23] >> Hello hi I'm Shashendu I work for Apple [1:28] uh in a product oriented research team [1:29] called video imaging and applications. [1:31] So like I think anything from um like [1:34] photos or cameras uh that has been like [1:37] developed uh by my team. Um and uh yeah [1:41] before that I used to be a grad student [1:43] at CMU and uh studying robotics and [1:46] computer science. [1:49] >> Good afternoon everyone. I'm air. I'm a [1:52] machine learning engineer at Optimx. I [1:55] work on building AI systems in home [1:58] deliveryarmacies. [1:59] So we work on automating pharmacy [2:03] with the with helping in increasing the [2:05] accuracy, reliability and also the [2:08] fulfillment times for our customers. So [2:10] I'm happy to be here and share my [2:12] insights. [2:14] >> Thank you guys. So I think let's begin [2:17] this session with um understanding the [2:20] history of computer vision. Uh I think [2:22] that'll be helpful for for a lot of [2:24] people in the audience. Now computer [2:26] vision has evolved. Uh in good old days [2:29] there used to be a lot of steerable [2:30] filters. Then we moved into [2:32] convolutional neural networks. Then we [2:34] moved into transformer models. And then [2:36] now we are moving into multi-mode [2:40] um transformer models. Models have [2:44] increased in sizes from being hundreds [2:46] of parameters to be billions of [2:48] parameters. So maybe I'll start with [2:51] Shumid. Um and given um shummed your [2:54] background in retail analytics. So how [2:57] has classic computer vision evolved [3:00] um from when you started all the way to [3:05] the deep learning era and and to today's [3:07] world and especially for things like [3:10] inventory protection, loss um [3:12] prevention, customer analytics. So how [3:14] has it evolved? [3:16] >> Right. Uh so [3:19] yeah so uh I have been working in the [3:22] computer vision space from 2014 just [3:25] when deep learning had started. So we're [3:27] using uh very common computer vision [3:30] algorithms like canny edge or watershed [3:33] or something like that wherein you could [3:35] just put some kind of rules and u if you [3:39] have some pattern specific to the rules [3:41] only then it would work and it would [3:43] never work apart from that you would put [3:44] some kind of a threshold. I think this [3:46] is when deep learning came into picture [3:48] wherein you could identify different [3:49] patterns like a dog cat um etc. Right? [3:53] So applying that into products. So [3:55] silarn has like huge SKS close to like [3:58] 20,000 SKs uh SKs meaning 20,000 [4:02] different items that's sold in a store [4:04] right. So how do you identify these and [4:06] then what are the potential advantages [4:08] of that right? for example in uh in the [4:11] checkout system uh like different form [4:13] factors uh we try to use computer vision [4:16] to solve those problems. U this is how [4:19] uh deep learning evolved and then it was [4:21] first training models and then slowly [4:23] going into like zeros short learning. [4:25] How do you solve problems without [4:26] training models with uh lower models or [4:30] sometimes no models and now getting into [4:32] gen AI space wherein the cost is so low [4:34] that you get charged only by the API [4:36] call. So um this is the journey that [4:39] I've seen uh in the last five six years. [4:43] >> Shushendu like you know Apple is very [4:44] big on ondevice machine learning these [4:47] days. Um um and this you know ranges [4:51] from doing next token prediction as you [4:54] are writing your SMS or iMessage all the [4:57] way to u illuminating a um a picture [5:02] that has been taken in a low light [5:04] condition. So there is a lot of machine [5:06] learning that Apple is um um utilizing [5:11] on a device constraint on a energy [5:13] constraint device. So [5:16] from your ondevice experience you know [5:19] again how has that classic lightweight [5:22] sort of models evolved into uh models [5:26] that not only take into account vision [5:29] but also text etc on an ondevice [5:32] setting. [5:34] Yeah, I think um classical computer [5:36] vision algorithms are still used and in [5:38] like a lot of places specifically like [5:40] edge contour detection um like they are [5:43] uh specifically used for like camera [5:46] autofocus or like video stabilization [5:48] these kind of things. Also classical [5:50] computer vision methods offer like a [5:52] very graceful fallback when um you know [5:55] the device is kind of busy doing [5:56] something else like most of these um [5:58] algorithms are currently like process in [6:01] like Google or like on iPhones they are [6:03] mostly like processed on specific [6:05] hardware like called A&Es or TPUs. So if [6:08] they are like busy these also like offer [6:10] like a very graceful u like fallback [6:12] options. Um also like uh from a product [6:16] point of view a lot uh of these devices [6:19] uh are dependent on the kind of hardware [6:21] that they have. Um so like u let's say [6:24] like u maybe like an iPhone from 2020 u [6:28] and it has a portrait mode. it'll just [6:30] focus on like person only but then it [6:32] has evolved into like a lot of different [6:34] things in like the current iteration of [6:36] the phone and um yeah and uh so like [6:42] utilizing these uh uh conventional [6:45] classical methods is also very helpful [6:48] in like complex AI systems too uh as an [6:50] input uh because you uh a lot of times [6:54] uh the images are kind of pre-processed [6:56] using these methods and then they are [6:57] processed using these uh like complex [6:59] algorithms. So yeah, I mean there's a [7:02] definite shift uh from these methods to [7:05] like the current uh datadriven methods [7:08] we are using but still like the previous [7:09] methods are pretty important. [7:11] >> Sounds good. So maybe I think you know [7:14] shifting the focus let's say from very [7:16] retail oriented um uh company like [7:21] 7-Eleven ondevice machine learning now [7:24] all the way to a regulated industry um [7:28] like in pharmaceutical operations. So [7:29] very much similar to um to JP Morgan who [7:33] is very very regulated in whatever we do [7:37] from I'm imagining like you know the [7:40] same sort of a constraints um are there [7:43] in the healthcare industry. So am I from [7:46] from your point of view how has the [7:49] traditional CV evolved? Um do you still [7:54] use it? is u if not like you know how do [7:57] you put together a system that we as a [8:01] customer of of um you know uh companies [8:05] like Optum um can say that it's you know [8:10] we are getting the best of the breed [8:12] >> right so classical computer vision [8:15] techniques as sidenu and sume mentioned [8:17] are a crucial piece even it's the same [8:20] for us as well so very specific problems [8:24] uh like problems where you want to solve [8:26] for explanability, reliability and [8:29] transparency. Uh these sort of scenarios [8:31] is where uh traditional techniques still [8:34] excel. Uh also when it comes to [8:39] time complexities, most traditional [8:41] methods have linear time complexities [8:44] compared to quadratic or exponential [8:47] time complexities in the recent attent [8:50] attention-based architectures. So this [8:53] makes it ideal for us to use it in [8:56] realtime systems where latency is in [8:58] requirement. So it sort of becomes a [9:00] no-brainer to try it out in situations u [9:03] where you want real time high throughput [9:05] high throughput solutions. A couple of [9:08] other places where we value it is where [9:13] we have space and budget constraints. So [9:16] most industrial settings you are equipp [9:18] equipped with cameras and [9:20] microcontrollers. So there is no GPU [9:23] access or cloud compute available. So [9:25] sort of you then are pigeonholed into [9:28] trying out traditional methods first and [9:31] seeing if they work for you. So [9:33] situations it just depends on what use [9:35] case you are using. Uh but I still feel [9:37] like u traditional methods have their [9:40] place. [9:41] Sounds good. Now again I think you know [9:43] when we um talk about the leap from [9:46] classical computer vision all the way to [9:49] transformers and and multimodel [9:52] um uh algorithms cost is a big thing [9:55] because um there aren't a lot of [9:58] companies who can throw in billions of [10:00] dollars to to train these super large [10:02] models right um so maybe I think [10:05] shishendu question to you is at Apple [10:09] what's your approach in balancing cost [10:11] constraints versus training these super [10:13] large models. Um because [10:17] it comes with a price. Um how do you [10:20] like imagine like when you're working in [10:22] a in a project like trading off one [10:25] versus the other? [10:27] >> I think cost constraint is something [10:30] that is only considered for the [10:32] inference a lot of times. I mean I don't [10:35] think Apple has any constraint of [10:36] budget. [10:38] >> Sure. But yeah, but uh I think the major [10:42] part of it is uh considering how we can [10:45] run these things things on devices and [10:48] there are like a lot of solutions uh for [10:50] it. Um of course like there's the ANA [10:52] which is just designed to make these [10:55] matrix computations way more faster than [10:57] a CPU or a GPU can do um in less amount [11:00] of time. Um and things are definitely [11:02] like progressing on that end. Um like [11:04] there are other companies too which are [11:06] developing on the same front. Um, [11:07] additionally there are uh tools like [11:09] called CoreML which offers um like it it [11:13] is basically like a bridge between your [11:16] model and uh like how to harness the [11:19] whole hardware ecosystem that uh Apple [11:21] offers. Um so it it contains like a lot [11:24] of different uh techniques for model [11:26] compression like quantization, pruning [11:28] these kind of things and um also it has [11:30] adaptive like dynamic um um like [11:34] protocols or like rules which kind of [11:37] route these u model layers to either [11:40] like CPUs, GPUs or A&Es. U that helps a [11:43] lot u um also like there's a it it all [11:47] operates like on a unified memory. So [11:49] like GPU, CPU, they all occupy the same [11:52] uh memory bus. So there's not like a lot [11:55] of time uh that is spent on you know [11:58] copying things from here uh to there. So [12:01] yeah, I mean like these things like [12:02] really um help to run things on device [12:06] but uh running like a whole LLM I think [12:10] it's pretty hard even now with all these [12:13] things and yeah people are making uh [12:15] developments on like these fields like [12:17] everywhere. [12:18] just looking forward to it. [12:20] >> So maybe coming back to you Shumid like [12:22] you know retail is a very cost-sensitive [12:25] um industry right. So how do you think [12:29] about like you know if you are going [12:31] into uh places like uh cashierless store [12:36] etc. [12:37] How are you trading off cost versus the [12:41] ROI of the um AI that you are [12:44] manufacturing? [12:46] >> Right. So for sure um cost plays a very [12:50] big role for us. Uh we can't really like [12:53] have like tons of GPUs because we do [12:55] have constraints especially uh when [12:58] you're dealing with stores uh you have [13:00] like connectivity issues and things like [13:01] that within these constraints you have [13:04] to come up with the solutions. So uh [13:07] classical solutions are the best because [13:09] sometimes you'll have to run uh at 30 [13:13] frames per second and if you go with [13:15] like very uh heavy uh heavier object [13:18] detection or more heavier models uh you [13:22] cannot get that 30 fps and then you get [13:25] a lag and then decision making is slow [13:28] etc. It frustrates the customer. So [13:31] where classical computer vision is [13:32] required we have to use that wherever it [13:35] cannot solve then you go with deep [13:36] learning. Again when you go with deep [13:38] learning you have to look at like how [13:40] sensitive is the situation. If it's [13:42] something like just uh if you miss a few [13:44] frames it's okay then you can apply that [13:47] and then in situations where you're okay [13:49] with time and you can do it offline you [13:52] can upload it to the cloud sometime [13:53] later as a batch then you can upload it [13:56] to the cloud. So it's like a trade-off. [13:58] um we don't have like a specific [14:01] solution that okay everything should go [14:03] to the cloud but optimizing cost is very [14:06] important for us right and I think Amaya [14:09] you are in in almost um in the middle in [14:13] the sense that not only pharma has to be [14:16] cost-ffective but also very reliable [14:19] right because um patients life you know [14:23] drugs etc are all related to that can [14:27] you share Is there an example where [14:30] either cost compute like you know cost [14:32] constraints or regulatory hurdle forced [14:35] you to rethink or simplify the solutions [14:39] um that you're working on. [14:40] >> Yes, absolutely. Uh there there are [14:43] times where cost compute does drive us [14:46] to think creatively and like figure out [14:49] solutions. So anytime you run into [14:51] problems, you don't want to throw more [14:53] GPUs or hardware at it. you want to make [14:56] solutions which are smarter and [14:58] affordable. So just to give you an [15:00] example um we're working on this real [15:03] time object detection pipeline which was [15:05] supposed to run at 60 frames per second [15:07] with the model we had trained at that [15:09] time we were only getting half of that. [15:12] So what you do in situations like that [15:14] is then you use tools like knowledge [15:16] distillation. So you can get like a [15:18] smaller model and then use a bigger [15:20] model to train your small model and then [15:22] have it run faster with the getting the [15:24] same same kind of accuracies. So that's [15:27] one way to do it also as uh Siden Shu [15:30] mentioned you can use optimiz [15:33] optimization techniques while [15:34] deployments like quantizations where you [15:37] sort of deploy the models at lower [15:39] inference so lower precision so that [15:42] increases your inference speeds uh you [15:44] can also use prunings so where you sort [15:47] of get rid of the redundant parameters [15:49] in deep neural nets and also Sumit Sumid [15:53] mentioned to add on to his cascaded [15:55] architecture is you have to figure out [15:57] if you have if the problems allows you [16:00] to then the complex cases go to a bigger [16:04] model and then the easier cases go to a [16:06] smaller model sort of then have a [16:08] trade-off between them. So it's just [16:10] about finding that right balance and [16:13] then figuring out what works for you. [16:15] >> Sounds good. Now every business is [16:18] unique. Um so we we were talking about [16:21] more the technology behind um solving a [16:24] given problem. Uh can you tell us maybe [16:27] am starting with you some of the use [16:29] cases that you work on in in in [16:32] particularly in computer vision? [16:34] >> Absolutely. [16:37] So we work in pharmacy space right. So [16:39] we do mail orderies. [16:42] So our significant portion of our [16:45] pharmacy we do like 100,000 [16:47] prescriptions every day. So to operate [16:49] at that scale you need vision systems to [16:52] run quality checks for you. So in case [16:55] where humans might miss things or the [16:58] vision systems help us catch it. So to [17:01] increase end to end throughput on our [17:03] lines we need to in integrate vision [17:05] systems like this into our system. [17:08] >> Got it. and and um maybe Schumid um [17:14] in terms of your use cases I'm guessing [17:17] it'll be mostly around checkout speed [17:19] inventory accuracy what are the other uh [17:23] use cases that u utilizes computer [17:26] vision in in retail [17:28] >> some of the use cases um that we use are [17:31] like some of the back office [17:32] optimization that we do uh it can go [17:35] around areas where you use media right [17:38] like video or images and then now you [17:41] have your multimodel LM you can actually [17:44] feed that get information from there and [17:46] then give it to a large language model [17:47] and get answers you you're trying to see [17:50] if we can identify uh problems within [17:54] like the marketing space in the merge [17:56] space because that's where we have a lot [17:58] of media we're trying to use [17:59] multimodelms for that and of course uh [18:02] product in itself is a huge thing for us [18:05] because there so many products and uh [18:08] some of the products you can scan with [18:10] barcode and for some you can't and then [18:12] if you can solve for those which are [18:14] high selling with computer vision uh [18:17] that removes the friction from the [18:19] customer so situations like those is [18:22] where we use computer vision [18:24] >> right and shishendu like for you I'm [18:28] guessing computer vision goes beyond [18:32] the kind of tech that we see on an [18:34] iPhone right u so what are these the use [18:37] cases that we as users we use it but we [18:40] don't know the the computer vision [18:42] happening behind the scene what are [18:44] these [18:46] >> uh [18:48] yeah I mean um face ID tracking that's a [18:51] pretty prominent use case and like when [18:54] you try to open uh your phone it does [18:56] like a live tracking of your face um [19:00] that I can think of that uh what else [19:02] there are like a bunch of different [19:03] features on like camera um spec [19:06] specifically like photos like it creates [19:08] like the whole memories thing. So there [19:10] goes like a lot of like uh deep learning [19:12] and computer vision behind that creating [19:14] those like salency maps and then [19:17] identifying people out of it and then [19:19] making these like people clusters and [19:21] then making like a whole movie out of [19:22] it. Um also like using these images to [19:26] generate to kind of get some sort of a [19:29] mood and then adding audio to it. U [19:31] these are like some basic like consumer [19:33] applications. And it can also go pretty [19:35] deep into um the creative space where uh [19:39] people use softwares like Final Cut Pro [19:43] or like um there's this new thing Black [19:45] Magagic Claps or something. Um so you [19:48] can create uh like green screens out of [19:50] frames and then you can replace it with [19:52] backgrounds which is like pretty uh [19:54] evident on Zoom or anything like that. [19:56] You can replace your background. So [19:58] these use cases um [20:02] what else? uh [20:05] depth. Uh like LA liar is also like a [20:07] pretty great uh like piece of technology [20:10] like specifically uh when you have a [20:12] portrait mode like when you try to use [20:14] portrait mode um it uses the camera and [20:18] also like the LAR to get the depth map [20:19] and then create that depth uh lens depth [20:22] effect. Uh [20:26] >> yeah um I can go on more like that. [20:29] >> Yeah sure. [20:30] Now I think you know that brings me to [20:33] to the next point that you know computer [20:37] vision I isn't the same as it used to be [20:40] 10 years ago because there is a lot of [20:42] mixed modality coming through the door [20:44] right you know people are are mixing [20:46] text and images videos [20:50] speech and all of that. So from from a [20:53] you know consumer devices point of view [20:56] what are the new um you know features or [21:00] problems that that you guys are thinking [21:02] around for an from an um ondevice [21:06] deployment perspective. [21:09] uh I mean I I can't really talk about [21:11] what we are thinking but in general like [21:13] I think [21:14] >> um the whole field is moving towards a [21:17] more um like holistic understanding of [21:20] whatever you are doing on your screen um [21:23] so some sort of like computer usage [21:25] where it has an understanding of um your [21:28] preferences or like your interactions [21:31] and then it can suggest things on top of [21:33] it for example like let's say you're [21:34] making a PowerPoint presentation and u [21:37] depending on the context that you have [21:38] on your screen. It might be able to like [21:40] suggest pictures that you can put in [21:42] your slides, something like that. Um or [21:45] let's say if you ask like an agent to [21:47] just describe u a picture that is on [21:50] your screen. Um it in also like when we [21:53] are talking about consumer devices, [21:55] there's this whole point of privacy. So [21:57] it's it's not really good to you know [21:59] send this pictures over to cloud because [22:01] it'll just hamper a lot of different [22:03] things. So instead of you know just [22:05] sending raw pixel data you can just uh [22:08] send like a semantic understanding [22:09] through like a language model and then [22:12] you can work on that to generate like a [22:14] description or something. So also like [22:16] in general like for you know touching [22:18] base on like creative workflows um you [22:21] can there might be like a day in the [22:23] near future where you could just uh you [22:25] know change the whole mood of an image [22:27] uh change the color toning just by [22:29] saying it and then it'll figure out [22:31] these workflows in the back end using [22:33] some like NAN is something that people [22:35] use a lot to create these ML u like [22:38] agentic workflows. So that might be [22:40] possible in the future. [22:41] >> Got it. And maybe Ahmed um over to you [22:45] in in I think Shishendu mentioned about [22:49] workflows. I'm guessing workflows become [22:52] much more acute in in [22:56] your industry. So what are the kind of [22:58] like you know we keep on talking about [23:01] LLM guided workflows etc. [23:04] um [23:06] what what are these use cases that [23:08] people are talking about specifically on [23:10] on your side of the world? [23:13] >> Yes, for sure. U combining computer [23:16] vision and LLMs would definitely be a [23:18] gamecher in terms of workflows inies. [23:23] I mean it is not an AI conference if we [23:24] don't mention AI agents, right? So we we [23:28] sort of figure out how do we move from [23:30] vision which is perception understanding [23:32] images into cognition where you [23:34] understand contexts. Um how do you make [23:37] smarter decisions faster is what it will [23:40] drive towards. So just to give you a few [23:43] examples of the workflows we could [23:45] potentially see it affecting is [23:48] prescription validations. So currently [23:51] we probably just read the we just have [23:54] OCR where we read what you see in the [23:56] image but then having like a semantic [23:58] understanding of what that prescription [24:01] is and then having sanity checks using [24:04] LLMs can can potentially add value to [24:07] value to the description validation [24:10] systems. When it comes to inventory [24:12] management as well, at some point you [24:15] would want to just point a camera at a [24:16] shelf and let the system figure out what [24:19] needs to be restocked or what what is in [24:22] abundance or how do how does it needs to [24:24] be restocked. So there are use cases [24:26] like that where combining vision and LLM [24:29] should help us also when it comes to [24:32] documentations. I'm sure like everyone [24:35] has used [24:37] email like email editings and at some [24:40] point edited documents using LLMs. So [24:43] how do we go from that into [24:46] putting images and videos into LLMs [24:49] which can potentially have P ph PHIP PII [24:52] informations in it. So how do you cover [24:55] how do you make the LLM understand the [24:57] regulations and compliance we have and [25:00] so I can see that gap currently and then [25:02] in the future it would be nice to see [25:03] how we cover that gap. [25:06] >> Sounds good. And I'm guessing Schum it [25:08] might be similar for you. Um I think um [25:12] Amay already mentioned that pointing uh [25:16] a camera to a shelf and counting things [25:19] um might be already old school in in in [25:21] retail right. So what are these uh [25:24] latest and greatest uh use cases that [25:27] brings in multimodality to the picture? [25:31] >> Yeah, I mean u apart from uh on the on [25:34] the storefront, the other areas are uh [25:37] how you can combine data across [25:40] different sources, right? uh like I said [25:42] like we have a lot of um media [25:44] information you can convert your media [25:47] information into text and then you have [25:49] these other data sources uh where you [25:51] have [25:53] different kinds of data like the [25:54] inventory data etc some of the marketing [25:56] data um some of the campaign data etc. [26:00] So you can take all of that data and [26:02] then u programmatically you can add that [26:04] in your prompt and then get your answer [26:06] and this is where the LM acts more like [26:09] an orchestrator and all this is possible [26:12] only because of the power of [26:14] multimodality because um if you just [26:16] look back just three years ago um the [26:19] wow factor was putting a bounding box [26:21] around the region of interest and today [26:23] it's moved from okay um I have a car in [26:27] front of me so you put a box and [26:29] behind the car if you have a tree the [26:31] multimodel LM can even tell that you [26:33] have a tree you have a car etc right so [26:36] that's why u multimodel LM is more [26:40] powerful compared to just using a deep [26:43] learning model and when it comes to cost [26:46] right it's significantly low because [26:47] you're paying per API cost or paying for [26:50] a set of tokens rather than um taking a [26:53] model hosting it somewhere running it [26:55] all the time maintaining the whole thing [26:57] etc So this is how the whole space is [27:00] evolving. [27:02] >> Got it. Now again you know um these days [27:05] no AI conference uh can happen without [27:08] the without talking about multi- aent [27:11] systems right. So in your case um you [27:16] spoke about multimodality. Now [27:18] multimodality can be across you know one [27:21] agent or multiple agents. How do you [27:23] like you know think about the these sort [27:26] of use cases or like you know multi- [27:28] aent [27:30] >> multimodal modality together? Yeah, like [27:34] slightly uh deviating from computer [27:36] vision a little bit here. Um multi- [27:39] aents as far as what we have seen is u [27:42] it's the ability for one large language [27:46] model to move the question to the right [27:49] agent uh understanding the kind of [27:51] question that the user has asked. Right? [27:54] Previously what we would do is we would [27:56] have like set of data and then you would [27:58] train a model like a logistic regression [28:00] model or an SVM model telling okay if [28:03] you ask this question predict it to like [28:05] this label or this label. Uh today even [28:08] if you don't have the data you can just [28:10] give it to your large language model and [28:12] it'll be like this wonderful router [28:14] which will take you to the right agent. [28:17] Right? This is where a multi- aent has [28:19] become so popular and it's just blown up [28:21] because without any data you can [28:24] literally get a solution which is [28:26] relatively accurate. So this is a place [28:28] where uh multi- aent systems are been [28:31] very popularly used [28:33] >> and and how how is the retail industry [28:35] thinking about you know injecting it in [28:38] in the day-to-day products. [28:42] So um I cannot specifically talk about [28:45] obviously the retail industry they have [28:47] like couple of components like they'll [28:49] have like the transaction information [28:51] they'll have like inventory information [28:52] they'll have like financial information [28:54] and then of course the media information [28:56] right like if you can create a platform [28:59] such that you have all of this data in [29:01] one place then you can build an LM on [29:04] top and then you can take all of this [29:06] data and then feed to the LLM to give [29:09] you an answer and then you can check [29:11] whether that even makes sense or not [29:13] because you have like text to SQL models [29:16] and there are like so many different [29:18] methods that you can use to basically [29:20] give all of your data in different [29:22] formats like maybe Excel file or images [29:25] everything into like normal English that [29:27] you can give it to an LM and it can [29:29] understand English [29:31] >> and um I think moving on to Shishandu [29:34] like what sort of a use cases do you [29:36] think um especially on an ondevice um uh [29:42] constraint area will multi- aent bring [29:44] to beer. [29:47] I mean one of the particular areas that [29:49] I'm very like personally interested like [29:51] you are from finance like uh like this [29:55] whole uh domain of agents have been made [29:59] for coding specifically [30:01] >> as cursor has been served but nothing as [30:03] such have been made for like [30:05] spreadsheets u in general and I feel [30:08] that is an area uh which will like [30:12] develop a lot in the next few years uh [30:15] because there's there's nobody like [30:17] who's kind of doing it and it requires [30:18] like this whole multimodal understanding [30:20] because like data is in some sort of [30:23] structure in sheets. It's just not very [30:24] random. Um and uh the changes that you [30:27] have to make inside these things have to [30:30] be like very objective like for coding [30:33] you can like code a single thing in many [30:35] different ways. You can create like [30:36] functions, you can create classes or [30:38] whatever you can create different files. [30:40] U I don't I think like things will be [30:42] like way more constrained in these [30:43] cases. So that's one obvious like use [30:46] case that I can see. Um yeah and like [30:50] these things run on devices like nobody [30:52] is uh using u like I think most people [30:55] just use Excel not Google sheets I [30:57] guess. [30:58] >> Uh and yeah the whole world runs on [31:00] that. [31:01] >> No and I think you know you brought a [31:03] very interesting point. So for financial [31:05] services like if you were to think about [31:08] the machine learning applications in in [31:10] financial services [31:12] I think it's a no-brainer if you can do [31:15] something for word excel powerpoint [31:17] >> so reading word documents generating [31:20] word documents [31:22] >> analyzing excel sheets and creating [31:25] excel sheets or creating understanding [31:28] powerpoint and creating powerpoints will [31:31] solve a majority of the problems that [31:34] the financial services industry um go [31:38] away with. So you're spot on and um [31:41] >> like people are work like there there is [31:43] like a lot of work on um like data [31:45] ingestion in general and then the last [31:47] part where you have like this whole [31:49] analysis in front of you and you want to [31:51] make like PowerPoint slides or just like [31:53] in general graphs or something u you can [31:55] do that but um the whole messy middle [31:57] work that's where I think uh things will [32:00] evolve. [32:01] >> Yeah. And and for those of you who [32:03] haven't done it because you know we we [32:05] have started to do um all of this it's a [32:08] non-trivial problem. You might think [32:10] that you know go as a human can you can [32:13] open up an Excel sheet and analyze it. [32:16] uh however as a machine as a LLM it's a [32:19] non-trivial problem because it's very [32:22] important to remember in regulated [32:23] industry like in in pharmaceuticals or [32:26] or in financial services you don't have [32:29] the luxury of being 95% accurate you [32:33] have to be either 100% accurate or not [32:36] so so that's why you know things which [32:40] might sound as a very nice you know MVP [32:42] problem like analyzing excel sheet is [32:45] very difficult to do in production. Um [32:49] so so we have our own stories around [32:52] around that. Um moving on a like you [32:55] know from your point of view how do you [32:57] think about you know a world where [33:01] multiple intelligence systems are [33:03] coordinating your pharma supply chain. [33:07] You know robots stacking on the shelf [33:09] and then you know cameras looking at it. [33:11] human beings coexisting in that [33:13] ecosystem while maintaining you know [33:15] traceability and compliance like how can [33:18] one think about it? Yes, absolutely. Oh, [33:22] majority of our team does work on [33:24] combining multiple intelligence systems [33:26] together. So the real tricky part there [33:29] is handling realtime communication [33:31] between these systems. So how do you h [33:34] sort of coordinate the work going from [33:37] system A to system B and make sure that [33:39] there is handoff of information at the [33:42] same time. So just to give you an [33:44] example uh we have order fulfillment [33:47] stations which combine robots which move [33:50] the products around and cameras to [33:53] visually inspect those products and then [33:55] analy analysis we run analysis on those [33:58] systems to track performances. So how do [34:01] you have like a central system which [34:03] coordinates all these different [34:05] individual systems together and then put [34:08] it put it together right so that is [34:09] where the challenge lies everything we [34:12] do is like audit ready by design so the [34:16] handoff the communication handoffs [34:18] between systems images timestamps [34:21] everything gets logged and from the time [34:25] the product coming into ouries and like [34:29] processed by our machines and [34:31] distributed to the customers. There is a [34:34] chain of custody which follows. So yeah, [34:37] essentially everything we do is like [34:39] tracked, traceable and recorded. [34:42] >> Sounds good. Um I know like I think we [34:45] are around you know 10 or so minutes um [34:49] u for this session. Um am I maybe [34:53] starting from you um for those building [34:56] careers right now in in let's say [34:59] regulated industry what would your [35:01] advice be like you know both for people [35:04] who want to pursue it technically and [35:06] also for people who' want to pursue it [35:08] from a non-technical side of the world. [35:11] Yeah, [35:12] >> absolutely. Uh I think couple of things, [35:15] right? You need to have like a problem [35:17] first mindset. So how do you usually the [35:21] problems remain the same where you're [35:22] track tackling for accuracy, [35:24] reliability, speed, right? So the [35:27] problems remain the same and then the [35:28] algorithms update over time. So how do [35:31] you stay updated with the algorithm [35:33] updates, right? So that's one technical [35:36] thing you need need to know. U couple of [35:40] non-technical [35:42] key points you need to be aware of I [35:44] feel is like clear simple communication [35:47] between teams. So when you talk to your [35:50] stakeholders and get gather the business [35:53] requirements uh you need to be able to [35:55] translate those into technical [35:57] requirements for your developers right [35:59] so you sort of have to be the leazison [36:01] between understanding the business [36:03] values and then driving the technical [36:06] aspects of the project. So having a [36:09] shared common language between your [36:11] stakeholders and your developers helps [36:13] there and then set set expectations on [36:17] things you're expecting out of the [36:19] project beforehand also helps make make [36:22] things easier. Um then secondly I would [36:26] say don't be disruptive when trying to [36:30] do trying to make changes in the system. [36:33] So a lot of our work involves like [36:36] humans in the loop. So you do not want [36:39] to go in on day one ch changing [36:42] disrupting their workflows. So sort of [36:44] how do you um seamlessly integrate into [36:47] the current systems. So that sort of [36:49] drives the impact and adoption from our [36:53] experience. So when we when we develop [36:56] systems we develop for humans first the [36:59] compliance second and machines third. [37:03] though I'm guessing it might be a [37:06] similar thought process as you like you [37:08] know would impart advice um to folks and [37:12] particularly the aspect that I would [37:14] want to know from you is you know there [37:15] is a resurgence of of robotics these [37:20] days a lot of like you know hardware [37:22] software codees moving on um day in day [37:26] out a lot of world models coming in [37:28] which is again the next generation of [37:30] multimodal [37:32] to its you know next iteration. [37:35] For people thinking about these sort of [37:37] domains, what would your advice be? What [37:39] are the things that that they really [37:41] need to crack on? [37:43] >> I think like one single greatest piece [37:46] of advice that I received and I think [37:49] like it can be applied to like [37:50] everything is that uh focusing on the [37:52] fundamentals rather than frameworks. So [37:55] just understanding the underlying [37:56] mathematics or how the algorithm works [37:59] that helps a lot and at the end of the [38:01] day like uh frameworks will keep on [38:04] changing like today it's transformers [38:05] tomorrow it's just something else like [38:07] maybe something else which is able to [38:08] like uh you know process um images like [38:11] 3D meshes all at once uh but at the core [38:14] like the understanding that's like [38:16] pretty important um the I think the [38:19] other thing is that uh most most of the [38:22] impactful uh like products that are [38:24] built are built with taking like cutting [38:27] edge research and thinking it from like [38:28] a consumer point of view and that's [38:30] exactly what happened with uh like [38:31] attention is all you need and so the [38:34] people like from open air they turned it [38:36] into like GPT which we all know uh yeah [38:39] and uh I think uh also like these days [38:44] including me a lot of people like are [38:46] VIP coding which is which is like a very [38:49] great place to be honestly because you [38:51] can build so many things on your own but [38:53] also I feel like understanding uh the [38:55] whole system architecture that's where [38:58] uh you should like put effort in because [39:00] like when there is a point where you [39:03] want to like scale it to like number of [39:05] users or scale it to like uh transition [39:07] it to like a different hardware uh like [39:09] yeah these things uh you have to like [39:11] kind of keep that in mind to uh [39:16] these changes. [39:17] >> Thank you. Um maybe taking a you know [39:20] other end of the coin um Shouid like as [39:24] you are hiring people right for your [39:26] group what do you want to look at? [39:30] >> Yeah like today things have changed a [39:33] lot any popular algorithm you can just [39:36] put it to any of the uh chat bots like [39:40] chat GBT or uh the perplexity or [39:43] whatever it'll just give you an answer. [39:46] I think um it's more about building the [39:49] algorithm, building the product, put it [39:52] out in front of the customer or whoever [39:55] in the wild, right? And then when it [39:58] does not work and when you take that hit [40:00] and when you go back and when you see [40:02] what's changing and then when you [40:05] understand the way people are [40:07] interacting with your product I think [40:10] that is where uh you have tremendous [40:12] learning and because what happens is [40:15] then you understand that okay this is [40:16] where the prediction has gone wrong or [40:18] this is where um it's giving like very [40:20] weird answers and then you go back and [40:22] you retrain on those right because [40:24] training a model is not difficult today [40:27] it's easy to do it and then when you [40:28] again put it out there and then when you [40:30] see that uptick I think that is a skill [40:33] that people should have because that [40:35] will actually tell you how you can [40:37] actually build systems that people can [40:39] actually use. [40:41] >> Thank you. I think um this is a um great [40:44] point because hiring machine learning [40:47] talent is non-trivial again um you know [40:49] I hired hundreds of people um there are [40:53] a lot of people you know who have the [40:55] right keywords on on the CV but the [40:58] actual time [41:00] emerges of a technical interview [41:02] everything falls apart right so I think [41:05] you know having people to not only know [41:07] the mathematics and the computer science [41:09] aspect of it but also the product aspect [41:11] of it because most companies would hire [41:15] talent who would be a value ad to the [41:18] top line or the bottom line you know [41:20] reducing the bottom line improving the [41:22] top line. So I think this is a very good [41:24] point that that you brought up. Um I [41:27] think we have questions like you know [41:29] time for one or two questions if anybody [41:31] in the audience so we have somebody [41:33] there. [41:39] One thing I didn't hear you guys talk [41:40] about, but I'm curious your perspective [41:42] on, we're moving some parts of our [41:45] workloads to large vision models. We're [41:47] also fine-tuning a large vision model [41:48] over like, you know, CNN kind of [41:50] architecture for um [41:54] I'll call them labeling problems. Um [41:58] my current understanding and my current [42:01] thinking is the barriers to entry for us [42:03] using large vision models more than [42:05] training our own, you know, old [42:08] traditional ML um is that they lack sort [42:11] of the world model stuff, right? They [42:13] lack the spatial understanding or lack [42:15] like certain pieces that don't allow us [42:16] to actually use them to their full [42:18] capacity. But I'm curious if you guys [42:19] see more workloads moving out of the, [42:22] you know, CNN kind of space towards the [42:25] LVM space. Like machine learner's job [42:27] might end up being fine-tuning an LVM [42:29] versus [42:31] training with labeled data a um set of [42:35] neural networks. But you guys are in [42:37] much different industries than I am. So [42:39] I'm curious if you agree. [42:45] Um so your question is like uh are you [42:48] seeing a world where people stop [42:50] training CNN's and move completely [42:52] towards VLMs? Uh yeah I mean like I [42:55] think it has been pretty much [42:56] established that they are better at like [42:59] understanding like spatial understanding [43:01] and also temporal understanding. It's [43:02] just the point of uh like can we run [43:04] these things on uh like a computer like [43:07] or or device like it's it's the [43:09] computational overhead is like huge. I [43:11] think if that is kind of solved. Yeah. I [43:14] mean I think we'll use them in most [43:16] cases. I think yeah. [43:19] >> So one other thing to add here is if [43:21] it's like day-to-day stuff like maybe a [43:24] tree or a car or even like a product, [43:27] right? I think in those situations maybe [43:29] a large language model can solve a [43:31] problem. But if it's like say in the [43:33] manufacturing sector you need to [43:36] identify some pipe which is broken [43:39] versus some pipe which is actually [43:40] right. I think in those situations you [43:43] should probably start with CNN model and [43:46] then slowly move towards LM because [43:49] that's when you'll also understand the [43:51] prompting part and etc. So it depends [43:54] completely on the domain. [43:58] We have another question there on the [44:00] other side. [44:09] >> Thank you. Do you guys see a possibility [44:12] of using attention like mechanism for um [44:15] let's say computer vision? Let's say you [44:18] not sure what you see in this case. You [44:20] have lower confidence. You can kind of [44:21] put attention almost like put more focus [44:23] on that region and get better results [44:25] for that particular part of the image. [44:30] >> Good lord. [44:34] >> Could you repeat your question? [44:39] >> Do you see guys uh possibility of using [44:41] attention-like mechanism? I mean [44:42] attention used in LLMs and transformers [44:44] a lot but let's say for computer vision [44:46] it's probably not as uh well researched [44:49] I would say but it's possible probably [44:51] to use the same level of or same kind of [44:53] anal analogical attention mechanism to [44:56] uh kind of focus your model on [44:58] particular part of the image which you [44:59] maybe not recognize very well and stuff [45:01] like this just to improve uh you know go [45:03] beyond 99% of you know object [45:06] identification [45:07] >> I think the question is uh can we use [45:10] the attention which we use in LLMs and [45:13] then transfer it into vision based [45:15] models. Yeah, I think uh that has [45:16] already happened with vision [45:18] transformers. Um we we see like go so we [45:22] have transformed from CNN's onto vision [45:24] transformers where you could use them uh [45:27] potentially to detect objects and stuff [45:29] like that. So that trend has already [45:31] yeah started. [45:33] >> Yeah. Before this whole transformers [45:35] things came in there used to be [45:37] something called salency maps which was [45:39] exactly that but for CNN's. So it tended [45:41] to like like let's say if you have an [45:43] image and if there's a subject like an [45:45] animal or something uh you could [45:46] generate like u a one pixel u like frame [45:50] which was exactly like it but it would [45:52] just tend to the regions which had uh [45:54] the most prominent um like things in [45:56] this case like an animal or something. [46:01] >> Thank you. Any further questions? [46:05] Okay. So thank you uh ever so much for [46:09] for being here and let's thank our [46:11] panelists here um Amay Shendu Shint um [46:16] thank you thanks a lot thank