Instacart Deploys Multimodal AI to Curb Catalog and Picking Errors

Generative AI for Grocery E-Commerce With Instacart — Ai4

Ai4

Instacart overcame brittle legacy classifiers and unsustainable frontier-model costs by deploying fine-tuned vision-language models with self-reflection loops across its 22-million-item catalog and real-time in-store fulfillment.

AI-assisted summary of Generative AI for Grocery E-Commerce With Instacart — original source. Check the original for full context.

Source published:

Detailed notes

## Background Published on YouTube by Ai4 on August 19, 2026, this talk features Pradeep Karuturi, a Machine Learning Engineer at Instacart. The presentation covers how Instacart uses multimodal Vision-Language Models (VLMs) to solve catalog intelligence and fulfillment challenges across its massive grocery catalog, detailing production strategies like self-reflection and distillation to optimize cost, latency, and accuracy.

## TL;DR At Instacart's scale of over 22 million catalog items and nearly 100,000 retail locations, legacy machine learning architectures—including single-task text classifiers, brittle OCR pipelines, and visual metric learning—struggled with incomplete retailer metadata and blurry in-store shopper photos. To bridge these gaps, engineering teams deployed multimodal Vision-Language Models (VLMs) that reason jointly across package imagery and text for both catalog attribute extraction and real-time fulfillment verification. However, productionizing VLMs brought severe hurdles with hallucinations, inference costs, model non-determinism, and pick-latency constraints. Instacart tackled these production bottlenecks through self-reflection verification loops, fine-tuning smaller specialized models to bake domain guidelines directly into weights, and investing heavily in robust golden datasets that enable autonomous agentic prompt tuning.

## Atmosphere & style The presentation carries a pragmatic, focused, and unvarnished engineering tone that avoids generic generative AI hype in favor of operational reality. Karuturi speaks with calm authority, methodically walking through concrete system designs grounded in messy edge cases like damaged barcodes, unlisted UPCs, and subtle packaging differences. It feels deeply practical and candid, tracing an engaging arc from massive commercial scale to the unglamorous, essential grind of curation loops, prompt shortening, and inference cost economics.

## Key takeaways - [00:04](https://www.youtube.com/watch?v=Quz16KujtZI&t=4s) Instacart operates at massive scale, reaching over 98% of US households with more than $40 billion in annualized Gross Transaction Value across 2,200 retail partners and 100,000 store locations. - [01:20](https://www.youtube.com/watch?v=Quz16KujtZI&t=80s) Generative AI touches the entire grocery customer journey, from enriching over 22 million catalog items and powering conversational shopping assistants to real-time visual fulfillment verification. - [03:03](https://www.youtube.com/watch?v=Quz16KujtZI&t=183s) Catalog intelligence is inherently multimodal because vital dietary, health, and nutritional attributes are printed directly on physical product packaging rather than supplied in retailer text metadata. - [04:40](https://www.youtube.com/watch?v=Quz16KujtZI&t=280s) In-store fulfillment relies on real-time visual verification from shopper mobile photos when barcode scanning fails due to damaged labels, loose produce, or catalog discrepancies. - [06:08](https://www.youtube.com/watch?v=Quz16KujtZI&t=368s) Pre-GenAI architectures—including single-task text classifiers, brittle OCR pipelines, and metric learning image embeddings—failed due to high maintenance overhead, noise sensitivity, and a lack of support for fuzzy business matching rules. - [08:33](https://www.youtube.com/watch?v=Quz16KujtZI&t=513s) Multimodal Visual Language Models (VLMs) unify visual and textual reasoning into a single framework capable of generalizing across long-tail grocery taxonomy using prompt-based business logic. - [09:35](https://www.youtube.com/watch?v=Quz16KujtZI&t=575s) Deploying VLMs in production introduces severe operational hurdles, including overconfident hallucinations, token costs across multi-million item catalogs, picking latency constraints, and a 6% to 7% non-deterministic decision flip rate. - [11:38](https://www.youtube.com/watch?v=Quz16KujtZI&t=698s) Instacart uses a self-reflection 'double-thinking' prompt loop that forces the VLM to verify its initial extraction against guidelines, significantly boosting precision before promoting data downstream. - [11:38](https://www.youtube.com/watch?v=Quz16KujtZI&t=698s) Fine-tuning smaller, specialized VLMs embeds complex grocery semantics directly into model weights, shrinking prompt payloads and delivering substantial cost and latency reductions compared to frontier models. - [13:03](https://www.youtube.com/watch?v=Quz16KujtZI&t=783s) Investing in high-quality golden datasets establishes dependable evaluation benchmarks and enables autonomous agentic loops to optimize prompts without human engineering intervention. - [14:33](https://www.youtube.com/watch?v=Quz16KujtZI&t=873s) Real-time out-of-stock predictions combine historical cyclical demand models with live, crowd-sourced availability reports streamed directly from in-store shoppers. - [14:33](https://www.youtube.com/watch?v=Quz16KujtZI&t=873s) Instacart applies a task-dependent model routing strategy: smaller fine-tuned models handle high-volume batch extraction and latency-sensitive picking, while frontier models are reserved for low-traffic edge cases.

## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=Quz16KujtZI&t=4s) Instacart's Scale and Ecosystem Overview

Instacart operates as the leading grocery e-commerce platform in the United States, reaching more than 98% of American households. The sheer operational scale powering the business is reflected in its financial and transactional footprint: reaching over $40 billion in annualized Gross Transaction Value (GTV), serving more than 26 million active customers who placed over 350 million orders over the past year across 2,200 retail partners. The platform provides consumers the flexibility to shop from nearly 100,000 brick-and-mortar store locations while offering earning opportunities for in-store personal shoppers who pick and deliver those orders.

- **Household Reach:** Over 98% of US households - **Annualized GTV:** More than $40 billion - **Order Volume:** Over 350 million orders placed in a single year - **Customer Base:** Over 26 million active customers - **Retail Footprint:** 2,200 retail partners representing nearly 100,000 store locations

---

[01:20](https://www.youtube.com/watch?v=Quz16KujtZI&t=80s) Broad Generative AI Applications Across the Grocery Journey

Generative AI touches virtually every critical stage of the grocery shopping workflow at Instacart. At the foundational level, GenAI models enrich the underlying product catalog by extracting, discovering, and healing structured attributes across a massive repository of more than 22 million unique items. Beyond catalog enrichment, large language models power search and discovery experiences through advanced query understanding, query rewriting, and personalized storefront merchandising—innovations documented in recent research publications presented at SIGIR and KDD.

```mermaid flowchart TD accTitle: Instacart GenAI Touchpoints subgraph PrePurchase["Discovery & Basket Building"] A["Catalog Intelligence\n(22M+ items enriched)"] --> B["Search & Query Rewriting"] B --> C["Personalized Merchandising\n(KDD/SIGIR Research)"] C --> D["Conversational Shopping Assistant\n(Recipe parsing & basket building)"] end subgraph PostPurchase["In-Store Fulfillment"] D --> E["In-Store Shopper Picking"] E --> F["Multimodal Fulfillment Intelligence\n(Visual verification of picks)"] end ```

The consumer experience also features a conversational shopping assistant designed to help customers parse complex cooking recipes, navigate dietary preferences, and automatically assemble shopping baskets. However, the most challenging technical frontiers sit at the two outer extremes of the funnel: catalog intelligence on the ingestion side, and fulfillment intelligence on the in-store shopper side. To address these, Instacart has turned heavily toward multimodal Large Language Models, commonly termed Visual Language Models (VLMs), to process rich text and noisy imagery jointly.

---

[03:03](https://www.youtube.com/watch?v=Quz16KujtZI&t=183s) Catalog Intelligence: Multimodal Attribute Extraction

When customers shop for groceries online, they rely heavily on structured product attributes to make critical health and dietary decisions. Shoppers regularly search for specific dietary constraints—asking whether a product is certified organic, gluten-free, low-fat, or what its exact sugar content and package sizing are. If these attributes are inaccurate or entirely missing from the catalog, customers either cannot find the items they want or risk purchasing goods that violate their dietary needs. Structured catalog data serves as "the substrate for every shopping decision online."

```mermaid flowchart LR accTitle: Catalog Multimodal Attribute Pipeline subgraph Input["Input Data"] T["Retailer Text Metadata\n(Often incomplete/noisy)"] I["Product Packaging Image\n(Contains nutrition, claims, badges)"] end subgraph VLM["Multimodal Processing"] P["VLM Pipeline"] end subgraph Output["Downstream Features"] B["Dietary Badges\n(e.g., 'Gluten-Free')"] V["Variant Grouping\n(Flavor/Size clusters)"] F["Search & Health Filters"] end T --> P I --> P P --> B P --> V P --> F ```

The fundamental obstacle in catalog intelligence is that retailer-provided text metadata is notoriously sparse and incomplete. Vital attributes—such as whether a snack is a "good source of fiber", "gluten-free", or "low in fat"—are almost never articulated in the text feed sent by the retailer; instead, they are printed solely on the physical product packaging. Consequently, attribute extraction is inherently multimodal. Instacart uses high-precision pipelines to extract these visual attributes, which in turn feed customer-facing features like dietary badges, variant clustering (such as organizing multiple chip flavors under a single parent item page), and precision health filters.

---

[04:40](https://www.youtube.com/watch?v=Quz16KujtZI&t=280s) Fulfillment Intelligence: Real-Time In-Store Product Verification

At the post-purchase end of the fulfillment funnel, shoppers navigate retail aisles to locate, pick, and scan items into customer orders. While barcode scanning is the standard validation mechanism, barcodes frequently fail in real-world retail environments due to damaged labels, unlisted UPCs, loose produce, or regional inventory differences. When barcodes fail, the platform must rely on photos taken directly by shoppers using their mobile devices to verify that the picked item matches what the customer requested.

```mermaid flowchart LR accTitle: Fulfillment Verification Flow subgraph Input["Shopper Verification Input"] CP["Customer Ordered Item\n(Catalog image + metadata)"] SP["Shopper In-Store Photo\n(Noisy, angled, shelf glare)"] end subgraph Inference["Real-Time Inference"] M["Product Matching Engine"] end subgraph Decision["Outcome"] OK["Approve Pick"] Nudge["Nudge Shopper\n(e.g., AA vs. AAA battery mismatch)"] end CP --> M SP --> M M --> OK M --> Nudge ```

This verification process represents a "classic product matching problem, but on steroids." Shopper photos captured in grocery store aisles are plagued by heavy background clutter, harsh store lighting, extreme angles, and motion blur. A common real-world scenario involves a customer ordering a package of AA batteries, while the shopper inadvertently grabs a AAA battery pack—a subtle visual difference easily missed by human oversight. Unlike catalog enrichment pipelines that run asynchronously offline, fulfillment verification operates synchronously in real time, making model inference latency a stringent operational constraint.

---

[06:08](https://www.youtube.com/watch?v=Quz16KujtZI&t=368s) The Limitations of Pre-GenAI Approaches

Prior to adopting generative AI, standard machine learning paradigms struggled severely across both catalog extraction and product matching. In catalog attribute extraction, engineering teams historically deployed single-task, text-based classification models—essentially training "one model per attribute." This architecture became an operational nightmare to maintain when scaling across hundreds of distinct attributes. Furthermore, attempts to bridge the visual gap using Optical Character Recognition (OCR) to extract raw packaging text before feeding it into NLP classifiers proved exceedingly brittle, as OCR errors cascaded downstream.

| Legacy Approach | Targeted Problem | Primary Failure Mode in Production | |:--- |:--- |:--- | | **Text-Based Classifiers** | Attribute Extraction | Required hundreds of bespoke models; failed completely when text feeds omitted visual packaging data. | | **OCR + Text Cleaning** | Image Attribute Extraction | Highly brittle extraction pipeline; OCR errors cascaded into attribute classification failures. | | **Image Embeddings (Metric Learning)** | In-Store Product Matching | Dual-encoder embedding cosine similarity collapsed when handling blurry, cluttered, angled in-store shopper photos. |

For product matching, teams previously leaned on deep metric learning by mapping catalog images and in-store shopper photos into a shared embedding space to compute cosine similarities. This approach collapsed under the noise of real-world shopper photos. Moreover, metric embeddings lacked the semantic nuance required for "fuzzy matching." Business logic often dictates controlled flexibility—such as allowing a 19-oz shampoo bottle as an acceptable fulfillment substitute for a requested 20-oz bottle rather than canceling the item outright.

Four shared technical challenges define both catalog and fulfillment applications: - **Inherently Multimodal:** Decisions require joint reasoning over both visual packaging cues and textual metadata. - **Extreme Precision Requirements:** False positives in fulfillment trigger millions of dollars in customer refunds, while incorrect catalog attributes destroy customer trust. - **High Data Noise:** Pipelines must ingest thousands of inconsistent retailer catalog schemas alongside low-quality, blurry in-store photos. - **Long-Tail Grocery Semantics:** Models must understand subtle domain-specific concepts, such as identifying A2 milk, lean cuts of meat, and retailer-specific private label brands.

---

[08:33](https://www.youtube.com/watch?v=Quz16KujtZI&t=513s) Multimodal Large Language Models as the Solution

Visual Language Models (VLMs) addressed these multifaceted constraints by unifying text and image reasoning within a single generalized framework. Rather than running disconnected OCR and classification steps, a single VLM ingests product imagery and metadata simultaneously, reasoning over visual features while directly adhering to complex prompt-based business rules. This allows the system to generalize effortlessly across long-tail grocery taxonomy without requiring custom-trained models for every niche subcategory.

```mermaid flowchart TD accTitle: Unified VLM Architecture for Both Use Cases subgraph VLM_Engine["Unified Multimodal VLM"] direction TB Prompt["System Prompt + Business Logic Rules\n(Enables exact & fuzzy matching rules)"] Vision["Visual Encoder"] LLM["Language Reasoning Backbone"] Prompt --- LLM Vision --- LLM end subgraph AttributeMode["Catalog Attribute Extraction"] A_Img["Product Package Image"] A_Txt["Catalog Title & Description"] A_Out["Structured JSON Attribute\n(e.g., gluten_free: true)"] A_Img --> Vision A_Txt --> LLM LLM --> A_Out end subgraph MatchMode["Fulfillment Verification"] M_Cat["Catalog Reference Image + Specs"] M_Shop["In-Store Shopper Photo"] M_Out["Match / No-Match / Substitute Decision"] M_Cat --> Vision M_Shop --> Vision LLM --> M_Out end ```

In catalog extraction workflows, the VLM receives the packaging image alongside item metadata and directly returns clean, typed attributes (such as validating gluten-free status). In fulfillment validation, the VLM evaluates the customer's ordered catalog reference alongside the noisy in-store shopper photograph, applying business rules directly in-prompt to execute precise or fuzzy match decisions.

---

[09:35](https://www.youtube.com/watch?v=Quz16KujtZI&t=575s) Production VLM Challenges and the Self-Reflection Pattern

Despite their superior reasoning capabilities, deploying VLMs into production environments introduced four major operational bottlenecks:

- **Hallucinations:** Models can exhibit overconfident hallucinations, such as incorrectly classifying a meat product as vegetarian, leading to severe health or dietary violations for consumers. - **Inference Cost:** Processing multiple high-resolution images across millions of catalog items consumes massive token volumes, making naive routing to top-tier frontier models financially unsustainable. - **Model Randomness / Non-Determinism:** Even with temperature set to zero and identical input parameters, ambiguous inputs cause model decisions to flip 6% to 7% of the time between retries, creating jarring user experiences where an identical item is rejected once and accepted seconds later. - **Real-Time Latency:** Multi-image reasoning calls can take multiple seconds, conflicting directly with the sub-second latency requirements of in-store mobile picking workflows.

```mermaid flowchart TD accTitle: VLM Self-Reflection Pattern subgraph Pass1["Initial Extraction"] In["Product Image + Metadata"] --> P1["VLM Pass 1:\nExtract Attribute"] P1 --> InitAns["Initial Attribute Output"] end subgraph Pass2["Reflection Step"] InitAns --> P2["VLM Pass 2:\nSelf-Reflection Prompt\n('Double check your answer')"] In -.-> P2 P2 --> RefinedAns["High-Precision Verified Output"] end ```

To counteract hallucinations and elevate precision without abandoning the architecture, Instacart implemented a self-reflection technique. Mirroring human critical thinking when prompted to double-check their reasoning, the system asks the VLM to review its initial extraction against the source inputs, forcing a structured verification step that measurably boosts output precision across ambiguous catalog items.

[11:38](https://www.youtube.com/watch?v=Quz16KujtZI&t=698s) Verification Loops and Fine-Tuning for Cost and Latency

To ensure high data quality in catalog intelligence, Instacart implements a self-reflection verification pattern before committing extracted information. Once the vision-language model (VLM) extracts an attribute value from product data and imagery, the system re-prompts the model: "Hey, given the attribute guideline, is the extracted value correct?" This "double-thinking" verification loop acts as a safeguard; only when the model confirms the extraction a second time is the attribute promoted downstream to production catalogs.

```mermaid flowchart LR accTitle: "VLM Attribute Extraction and Self-Reflection Loop" subgraph Extraction["Attribute Extraction"] A["Product Image & Text"] --> B["Frontier / Fine-Tuned VLM"] B --> C["Candidate Attribute Value"] end subgraph Verification["Double-Thinking Loop"] C --> D{"VLM Self-Reflection<br/>Checks Guidelines"} D -- "Confirmed Correct" --> E["Downstream Systems"] D -- "Mismatch / Flagged" --> F["Discard or Re-evaluate"] end ```

Deploying massive frontier VLMs across an entire catalog introduces significant cost and latency hurdles, particularly for real-time user experiences. To address this, Instacart creates high-quality datasets using a hybrid combination of human annotators and LLM-assisted labelers, then fine-tunes smaller, specialized VLMs. This strategy yields a dual benefit:

1. **Smaller Model Footprint:** The distilled, smaller model runs significantly faster and at a fraction of the inference cost compared to frontier models. 2. **Simplified Prompts:** For ambiguous domains where instruction prompts initially spanned tens of pages, the fine-tuning process bakes esoteric grocery concepts directly into the model's weights. Because complex retail definitions no longer need to be injected into every single API call, prompt payloads shrink dramatically, creating an additional layer of latency and cost savings.

[13:03](https://www.youtube.com/watch?v=Quz16KujtZI&t=783s) The Unglamorous Lesson: Golden Datasets and Autonomous Agentic Loops

A critical engineering takeaway is the need to heavily invest in golden datasets and rigorous data labeling workflows. Grocery catalog intelligence frequently encounters severe edge cases in product matching: noisy product imagery, multi-attribute comparisons across brand, size, flavor, and dietary restrictions, and instances where the on-package text directly contradicts the packaging imagery. In these scenarios, even experienced human annotators disagree or introduce labeling errors.

Without a reliable ground-truth golden dataset, engineering teams end up wasting time debugging conflicting labeling guidelines and data discrepancies rather than tuning prompts. Once a robust golden dataset is established, downstream stakeholders can trust evaluation metrics. Furthermore, high-quality golden benchmarks enable autonomous agentic optimization loops: automated systems can iteratively evaluate and tune prompt performance against the benchmark without human intervention, drastically lowering engineering overhead.

- Product matching requires reconciling conflicting signals across image visuals and text descriptions across dimensions like brand, package size, flavor profiles, and dietary certifications. - Clean ground-truth datasets allow evaluation metrics to serve as authoritative quality gates for downstream consumers. - Reliable benchmarks enable fully autonomous agentic prompt-tuning loops that continuously optimize model accuracy without human engineering in the loop.

[14:33](https://www.youtube.com/watch?v=Quz16KujtZI&t=873s) Q&A: Predicting Inventory Out-of-Stocks and Model Selection Strategies

During the audience question-and-answer session, an attendee operating a 1,500-SKU retail business asked how Instacart manages stock thresholds and substitutions given the uncertainty of retailer inventory feeds. Rather than relying purely on static retailer inventory figures, Instacart deploys predictive machine learning algorithms that factor in cyclical historical sales patterns alongside real-time data streaming from shoppers inside the stores. If multiple shoppers report back that a specific SKU is missing from shelves, the platform aggregates that signal, recalculates availability probability, and proactively marks the item as out of stock on the consumer app to prevent fulfillment failures and customer dissatisfaction.

```mermaid flowchart TD accTitle: "Real-Time Inventory Prediction and Availability Pipeline" subgraph Signals["Inventory Input Signals"] A["Retailer Feed Data"] B["Historical Cyclical Demand ML"] C["In-Store Shopper Real-Time Reports"] end subgraph Intelligence["Aggregation Engine"] A --> D["Real-Time ML Inventory Aggregator"] B --> D C --> D end subgraph Actions["Consumer Experience"] D --> E{"Item Available?"} E -- "High Confidence" --> F["Visible in App for Purchase"] E -- "Low Confidence / Out of Stock" --> G["Mark Out of Stock / Trigger Substitution"] end ```

When asked about model selection trade-offs—such as open-source versus closed proprietary models, inference serving harnesses, and post-deployment economics—the selection process is strictly task-dependent:

- **Latency-Sensitive Paths:** Real-time customer-facing workflows tolerate slight accuracy trade-offs in exchange for low-latency responses, favoring smaller fine-tuned models. - **Low-Traffic Edge Cases:** Niche workflows with minimal request volume can afford to route traffic to the most capable, frontier models regardless of per-token costs. - **High-Volume Catalog Ingestion:** For catalog-wide tasks like bulk attribute extraction, routing all requests to top-tier frontier models is financially non-viable because "the cost is going to be outrageous." These batch pipelines rely on smaller, fine-tuned models optimized specifically for throughput and cost efficiency.

## Notable quotes > [03:03](https://www.youtube.com/watch?v=Quz16KujtZI&t=183s) the substrate for every shopping decision online — Pradeep Karuturi

> [04:40](https://www.youtube.com/watch?v=Quz16KujtZI&t=280s) classic product matching problem, but on steroids — Pradeep Karuturi

> [06:08](https://www.youtube.com/watch?v=Quz16KujtZI&t=368s) one model per attribute — Pradeep Karuturi

> [11:38](https://www.youtube.com/watch?v=Quz16KujtZI&t=698s) Hey, given the attribute guideline, is the extracted value correct? — Pradeep Karuturi

> [14:33](https://www.youtube.com/watch?v=Quz16KujtZI&t=873s) the cost is going to be outrageous — Pradeep Karuturi

## Mentioned Pradeep Karuturi, Instacart, Ai4, SIGIR, KDD

## Video description Pradeep Karuturi - Machine Learning Engineer, Instacart

In this talk, I will illustrate how Instacart leverages multimodal Large Language Models (VLMs) to transform grocery e-commerce by solving complex challenges in catalog intelligence and fulfillment. Traditional text-based or embedding-based models struggle with the noisy, real-world data inherent in grocery retail. I will explain how we successfully deployed VLMs to handle multimodal inputs, drastically improving product matching and attribute extraction across our massive catalog. You will also learn about our practical strategies for productionizing these models, including self-reflection techniques to mitigate hallucinations and knowledge distillation to optimize for cost and latency.

[Source](https://www.youtube.com/watch?v=Quz16KujtZI)

中文简报

Instacart部署多模态AI以减少目录与拣选错误 Instacart通过在2200万种商品目录和实时店内履约中部署带有自我反思循环的微调视觉语言模型，克服了传统分类器的脆弱性以及前沿模型高昂不可持续的成本问题。 在Instacart拥有超2200万种目录商品和近10万家零售门店的业务规模下，包括单任务文本分类器、脆弱的OCR流程和视觉度量学习在内的传统机器学习架构，难以应对不完整的零售商元数据以及店内代购员拍摄的模糊照片。为弥补这些缺陷，工程团队部署了多模态视觉语言模型（VLM），结合包装图像与文本进行联合推理，用于目录属性提取和实时履约验证。然而，VLM的生产落地也带来了幻觉、推理成本、模型非确定性以及拣选延迟限制等严峻挑战。Instacart通过自我反思验证循环、微调较小的专用模型以将领域规范直接融入权重，并大力投资构建支持自主智能体提示词调优的稳健黄金数据集，成功解决了这些生产瓶颈。 Instacart业务规模庞大，覆盖超过98%的美国家庭，在2200多家零售合作伙伴和10万家门店中实现了超过400亿美元的年化商品交易总额（GTV）。 (0:04) 生成式AI贯穿整个生鲜购物流程，从丰富超2200万种目录商品、驱动对话式购物助手，到实现实时视觉履约验证。 (1:20) 目录智能本质上是多模态的，因为关键的饮食、健康和营养属性直接印在实体商品包装上，而非由零售商文本元数据提供。 (3:03) 当因标签损坏、散装生鲜或目录不一致导致条形码扫描失败时，店内履约依赖代购员手机拍摄的照片进行实时视觉验证。 (4:40) 生成式AI之前的传统架构——包括单任务文本分类器、脆弱的OCR流程以及度量学习图像嵌入——因高昂的维护成本、对噪声敏感以及不支持模糊业务匹配规则而难以奏效。 (6:08)

Reading summary

Instacart applies multimodal Visual Language Models (VLMs) to resolve critical operational bottlenecks at the extremes of its grocery e-commerce pipeline: incomplete catalog metadata ingestion and noisy, real-time in-store fulfillment verification. To operate reliably at scale, the platform pairs these models with self-reflection verification loops, specialized fine-tuning, and robust golden datasets. Multimodal Demands in Cataloging and Fulfillment (3:03) Instacart operates across 2,200 retail partners and over 26 million active customers, generating more than $40 billion in annualized Gross Transaction Value. At this scale, the platform encounters acute data quality challenges at both the ingestion and fulfillment stages. Online grocery catalog metadata provided by retailers is notoriously incomplete; crucial dietary attributes—such as gluten-free claims, sugar content, or fiber levels—rarely appear in retailer text feeds and exist only on physical product packaging. Extracting these visual cues across 22 million catalog items is essential for powering search filters and dietary badges. At the fulfillment stage, shoppers picking orders in stores face frequent barcode scan failures caused by damaged labels, missing UPCs, or unpackaged produce. To verify items, shoppers submit smartphone photos, which present severe operational challenges: motion blur, glare, sharp angles, and visual clutter. Verifying subtle distinctions in real time—such as distinguishing AA from AAA batteries—requires high-precision multimodal reasoning under strict latency constraints to prevent costly item mispicks and refunds. Legacy Architectural Limits and Unified VLMs (6:08) Before adopting generative AI, Instacart deployed single-task, text-based classifiers that required maintaining hundreds of bespoke models for individual attributes. Attempts to process packaging imagery via Optical Character Recognition (OCR) proved brittle, as optical reading errors cascaded directly into downstream NLP classifiers. For fulfillment verification, metric learning dual-encoders mapped catalog and shopper photos into shared embedding spaces, but cosine similarity collapsed under noisy in-store photography and could not accommodate fuzzy substitutions, such as approving a 19-ounce bottle when a 20-ounce item was requested. Visual Language Models unified text and visual processing into a single framework. By ingesting product packaging images and text metadata concurrently, VLMs apply system-prompted business logic directly to visual inputs. This generalized architecture enables the system to handle long-tail grocery taxonomy, such as recognizing lean meat cuts or A2 milk, without requiring custom classifiers for every category. Production Safeguards Against Hallucination and Latency (9:35) Deploying frontier VLMs into production revealed significant operational hurdles: dangerous hallucinations (such as labeling meat products as vegetarian), multi-second latencies incompatible with real-time picking, high inference costs across millions of items, and non-deterministic behavior where identical ambiguous inputs produced outcome flips 6% to 7% of the time even at zero temperature. To raise accuracy, Instacart implemented a self-reflection verification loop: after an initial attribute extraction, the VLM is re-prompted to verify its output against predefined guidelines, committing data downstream only upon confirmation. To mitigate latency and cost, the platform distills high-volume tasks into smaller, specialized VLMs trained on hybrid human- and LLM-annotated data. Fine-tuning embeds complex grocery taxonomies directly into model weights, shortening prompt lengths and significantly reducing token usage and response times. Golden Benchmarks, Agentic Optimization, and Routing (13:03) Instacart highlights ground-truth golden datasets as an essential prerequisite for managing contradictory edge cases where packaging imagery conflicts with retailer text descriptions. High-quality benchmarks provide reliable quality gates and enable autonomous agentic loops that continuously evaluate and tune prompts without human intervention. Model deployment follows workload economics: low-traffic edge cases use frontier models, while latency-critical and high-volume catalog pipelines rely on smaller fine-tuned models. Additionally, to overcome static retailer inventory feeds, Instacart deploys predictive machine learning that synthesizes historical demand cycles with real-time shelf reports from active shoppers, automatically marking items out of stock on the consumer app when multiple pickers report them missing. While visual language models resolve long-standing limitations in catalog enrichment and in-store verification, deploying them reliably requires structural guardrails against non-determinism, task-specific model distillation to contain inference costs, and real-time crowd signals to compensate for imperfect retailer inventory data.

精炼阅读摘要

Instacart在覆盖全美超过98%家庭的超大规模生鲜电商网络中，将多模态视觉语言模型（VLM）置于商品目录智能与店内即时履约的核心，以攻克传统机器学习在处理稀疏元数据、嘈杂线下图像及严苛业务规则时的系统性失效。 业务规模与全链路生成式AI布局 (0:04) Instacart年化商品交易总额（GTV）超过400亿美元，连接2600多万活跃用户、2200家零售合作伙伴的近10万家线下门店，过去一年处理了超过3.5亿笔订单。在此庞大生态中，生成式AI贯穿消费者全购物旅程：底层用于超过2200万款独特商品目录的结构化属性提取、发现与修复；在上游搜索与发现场景，大语言模型赋能查询理解、查询重写与个性化货架推荐，相关成果发表于SIGIR与KDD会议；面向消费者端，对话式购物助手则用于解析复杂烹饪食谱并自动组装购物车。 技术上最具挑战性的两端位于业务漏斗的最外侧：数据接入端的异步商品目录智能，以及代购员在实体店拣货端的同步履约智能。为应对这两个极端场景，Instacart已将工程重心转向能联合处理文本与嘈杂图像的多模态大型模型。 目录稀疏性与店内即时校验痛点 (3:03) 在目录端，消费者高度依赖有机、无麸质、低脂、糖分含量等结构化属性作出健康与饮食决策。然而，零售商提供的原始文本元数据普遍残缺，“富含膳食纤维”或“无麸质”等关键标签极少包含在纯文本数据流中，而是直接印刷在商品实体包装上。若缺乏精准提取，消费者将无法检索到合适商品，甚至面临违反饮食禁忌的健康风险。 在后端的履约场景中，代购员依靠扫码完成拣货，但条形码常因标签磨损、未收录通用产品代码（UPC）或散装生鲜而失效。此时系统必须通过代购员手持设备拍摄的照片进行实时核对。店内照片常伴随货架反光、运动模糊、遮挡与倾斜角度，且极易出现细微差错（如将AA电池与AAA电池混淆）。与离线异步运行的目录提取不同，履约图片匹配是强依赖亚秒级低延迟的实时同步任务。 传统架构失效与统一VLM解法 (6:08) 在引入生成式AI之前，传统机器学习在目录提取与匹配上面临瓶颈。在目录端，工程团队过去为每个属性单独训练单任务文本分类模型，维护数百个独立模型极为繁琐；而先采用光学字符识别（OCR）提取文字再接自然语言分类器的方案十分脆弱，OCR识别错误会直接引发下游分类级联失效。在履约端，基于深度度量学习的图像嵌入在面对嘈杂实拍图时余弦相似度失效，且无法实现模糊匹配的业务逻辑（例如允许用19盎司洗发水替代缺货的20盎司商品）。 视觉语言模型（VLM）通过统合文本与图像推理突破了上述限制。单个模型能够同时吸收包装图片与商品元数据，并在系统提示词中直接嵌入业务规则。这不仅消除了分立的OCR流程与对数百个小模型的维护需求，还能在无须针对细分长尾品类逐一训练模型的前提下，泛化处理复杂的生鲜食品长尾语义。 生产瓶颈、自我反思回路与模型微调 (9:35) VLM在生产部署中暴露出四大工程瓶颈：将肉类误判为素食等危及合规的确定性幻觉；数百万件商品高分辨率图像产生的高昂Token推理成本；多图像推理耗时数秒而无法满足移动端秒级响应；以及即使在零温度系数（Temperature=0）下，模型面对模糊输入仍有6%至7%的非确定性判断翻转。这些翻转会导致代购员在短时间内对同一商品重复校验时得到冲突结果。 为抑制幻觉并提升确定性，Instacart构建了“自我反思（Self-Reflection）”验证回路：在初次提取属性后，系统自动触发追问让模型比对指南进行二次核对，只有两次确认一致才写入生产目录。针对延迟与成本，团队利用人工与LLM协同标注的数据集对较小规模的专用VLM进行微调。微调不仅使小模型推理速度更快、计算成本更低，更将原先长达数十页的提示词规则直接内化到模型权重中，大幅精简了每次请求的Prompt负载。 黄金基准驱动与分层部署经济学 (13:03) 生鲜零售中经常出现包装图文矛盾、多重属性交叉等复杂边缘案例，甚至资深人工标注员也会产生分歧。Instacart的核心工程经验在于优先构建高置信度的“黄金数据集（Golden Dataset）”。统一的真值基准不仅能作为下游系统的质量门禁，还支持自主智能体闭环（Autonomous Agentic Loops）在无需人工干预的情况下自动评估与迭代微调提示词。 在模型选型与部署经济学上，Instacart依据具体任务类型分流：高并发的全量商品目录提取若使用顶级前沿模型将导致财务成本失控，因而主要采用高吞吐、低成本的微调小模型；延迟敏感的履约实时路径优先采用紧凑型模型；低流量的长尾极端案例才交由顶级通用模型处理。此外，针对零售商静态库存数据的不确定性，系统结合历史周期性销量模型与代购员店内实时的缺货上报，动态预测货架可用概率并在前端提早干预。 Instacart的实践表明，通用前沿大模型无法直接以原生态部署于严苛的生鲜电商生产环境，必须配合自我反思机制、小模型蒸馏微调和黄金数据集工程化，在成本、延迟与确定性之间取得工程平衡。然而，在复杂包装图文存在冲突时如何进一步消除非确定性波动，以及在极致低延迟限制下持续提升多模态推理鲁棒性，仍是持续演进的核心挑战。

Detailed notes — zh

## Background 该内容由行业会议组织者 Ai4 于 2026 年 8 月 19 日在 YouTube 平台发布。视频记录了 Instacart 机器学习工程师 Pradeep Karuturi 的主题演讲，主要分享 Instacart 如何运用多模态大模型（VLM）解决杂货电商中高噪声商品目录智能治理与实时履约核验的现实挑战，以及落地中的自我反思和知识蒸馏策略。 ## TL;DR Instacart 依托多模态大模型（VLM）应对杂货电商中复杂的商品属性抽取与店内代购实时匹配难题，攻克了传统纯文本模型和图像嵌入无法处理的强噪声与长尾概念瓶颈。面对模型落地时的幻觉、推理延迟和高昂算力成本，团队引入“双重思考（Double-thinking）”自我反思机制进行闭环校验，并通过微调轻量化小模型内化专业业务规则。同时，团队强调了构建高标准“黄金测试集”对于驱动 Agentic 自动化调优的核心价值，并展示了如何结合代购员实时反馈与机器学习预测解决线下库存动态波动的难题。 ## Atmosphere & style 整场演讲呈现出极其务实、严谨且偏重工程落地的技术分享基调。演讲者没有停留在生成式 AI 的宏大叙事上，而是直面杂货零售中模糊图片、条码损坏、多维度标注分歧等“不光鲜（unglamorous）”的现实泥潭，语调冷静而坦诚。现场问答互动切中库存波动与推理成本等工业界命脉问题，观众仿佛置身于一线大厂工程架构复盘现场，能深刻体会到算法团队在精度、延迟与天文数字算力账单之间反复权衡的硬核极客氛围。 ## Key takeaways - [00:04](https://www.youtube.com/watch?v=Quz16KujtZI&t=4s) Instacart 业务规模庞大（年化 GTV 超 400 亿美元，管理 2200 万+ 商品），生成式 AI 已全面渗透至目录治理、搜索推荐以及线下代购履约三大核心环节。 - [03:03](https://www.youtube.com/watch?v=Quz16KujtZI&t=183s) 杂货商品的大量核心特性（如有机、低脂、无麸质等）往往仅以图标或文字印于包装，使得属性提取在本质上必须依赖多模态信息而非纯文本。 - [03:03](https://www.youtube.com/watch?v=Quz16KujtZI&t=183s) 店内履约侧核验面临条码损坏、现场光线杂乱、拍摄模糊等强噪声挑战，系统需要在严苛的秒级低延迟下对代购员拿取的商品进行实时校验纠错。 - [06:08](https://www.youtube.com/watch?v=Quz16KujtZI&t=368s) 传统单属性文本分类与图像向量嵌入方案在维护成本、抗噪鲁棒性以及支持“模糊匹配”（如缺货时规格微调）方面均遭遇了难以逾越的瓶颈。 - [06:08](https://www.youtube.com/watch?v=Quz16KujtZI&t=368s) 多模态视觉语言模型（VLM）兼具联合解析图文、理解长尾零售概念以及在 Prompt 中灵活融入业务模糊规则的能力，成为破解难题的关键架构。 - [09:27](https://www.youtube.com/watch?v=Quz16KujtZI&t=567s) VLM 实际投入生产面临幻觉、极高算力成本、高推理延迟以及在相同输入下仍有 6% 至 7% 判定翻转的随机性问题。 - [11:38](https://www.youtube.com/watch?v=Quz16KujtZI&t=698s) 引入“双重思考”（Double-Thinking）自我反思机制，在初次抽取后促使模型对照规范二次核对，有效规避了关键属性的致命幻觉。 - [11:38](https://www.youtube.com/watch?v=Quz16KujtZI&t=698s) 通过高质量标注样本微调轻量化 VLM，不仅大幅削减了推理延迟和 API 成本，还让小模型直接内化了数十页复杂 Prompt 中的杂货专业领域知识。 - [13:03](https://www.youtube.com/watch?v=Quz16KujtZI&t=783s) 重金建设“黄金数据集”是解决杂货多维度图文冲突的基石，并由此打通了全自动迭代 Prompt 与指标评测的 Agentic 调优闭环。 - [14:33](https://www.youtube.com/watch?v=Quz16KujtZI&t=873s) 库存管理打破对零售商静态接口的单一依赖，结合周期性 ML 预测与一线代购员实时缺货反馈动态感知库存，前置拦截无货商品加购。 - [16:09](https://www.youtube.com/watch?v=Quz16KujtZI&t=969s) 模型选型高度取决于业务权衡：全量属性抽取坚决推行轻量微调以防 API 成本爆炸，实时交互适度让渡精度换取低延迟，低频长尾任务则直接调用顶级前沿模型。 ## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=Quz16KujtZI&t=4s) Instacart 业务概览与生成式 AI 应用全景 Instacart 作为美国领先的杂货电商平台，业务覆盖超过 98% 的全美家庭。根据最近一个季度的财报数据，平台的年化商品交易总额（GTV）已突破 400 亿美元；在过去一年中，超过 2600 万名活跃用户在平台上的 2200 家零售商处累计完成了超过 3.5 亿笔订单。平台不仅为消费者提供了跨越近 10 万家实体门市的选购便利，同时也为在店内执行拣选与配送的代购员（Shoppers）创造了高度灵活的增收机会。 在如此庞大的业务体量下，生成式 AI（GenAI）的应用已渗透到杂货电商的各个关键环节。最基础也是最核心的应用是商品目录（Catalog）的治理与丰富化——利用大模型跨越 2200 多万种独立商品（Unique Items）进行结构化属性的提取、发现与自动修复（Healing）。此外，生成式 AI 还广泛应用于搜索与个性化推荐场景，例如查询意图理解（Query Understanding）与改写。在用户交互层面，平台推出了对话式购物助手，帮助用户解析复杂食谱、一键加购并根据个人饮食偏好浏览商品。当订单生成后，线下代购员进店履约履约（Fulfillment）环节同样依托多模态大模型（VLM）进行智能校验。 - **年化 GTV**：超过 400 亿美元 - **用户与订单规模**：过去 1 年内 2600 万+ 用户完成 3.5 亿+ 订单 - **零售合作生态**：连接 2200 家零售商、近 10 万家线下门店 - **商品库量级**：管理超过 2200 万种独立商品 ```mermaid flowchart LR accTitle: Instacart 生成式 AI 应用全景 subgraph Catalog["商品目录智能 (Catalog)"] A["属性提取与修复<br/>22M+ 独特商品"] end subgraph Discovery["搜索与个性化 (Discovery)"] B["查询意图理解 / 改写"] C["个性化店面陈列"] D["对话式食谱助手"] end subgraph Fulfillment["履约智能 (Fulfillment)"] E["店内代购实时核验<br/>多模态图像匹配"] end Catalog --> Discovery --> Fulfillment ``` --- ### [03:03](https://www.youtube.com/watch?v=Quz16KujtZI&t=183s) 业务核心两端：商品属性提取与履约核验的现实挑战 在杂货电商业务的两端，商品目录智能与店内履约智能面临着极具代表性的技术挑战。在前端目录侧，消费者在购买食品杂货时非常关注具体属性：例如产品是否为有机（Organic）、含糖量多少、是否无麸质（Gluten-Free）以及具体净重规格。然而，合作零售商提供的纯文本数据往往缺失这些关键信息。许多核心健康卖点（如“富含纤维”、“低脂”）仅以微小图标或说明文字印在商品外包装上，必须通过商品包装图像才能获取，这使得属性提取本质上是一个多模态问题。这些高精度的结构化属性不仅用于打上特性标签，还驱动着相似口味归类的变体聚合（Variants）、多维度筛选过滤以及健康选品建议。 在后端的履约侧，代购员进店拣货时通常依赖条形码扫描进行验证。但在条码损坏、更换或无法扫描等诸多异常情况下，系统必须依赖代购员现场拍摄的照片进行校验。由于店内光线不佳、角度刁钻、背景杂乱甚至画面模糊，代购员上传的照片充满了极高的现实噪声。例如，顾客订购了 AA（5号）电池包，代购员因视觉疏忽实际拿取了 AAA（7号）电池包；系统必须通过图像识别并即时提示代购员修正。由于履约核验发生在代购员操作当下，这属于严苛的实时场景，对推理延迟有着极高要求。 ```mermaid flowchart TD accTitle: 商品目录提取与履约核验双端对比 subgraph CatalogSide["目录端：属性提取 (Offline)"] C1["零售商文本 + 包装图片"] --> C2["提取高精度属性<br/>(无麸质 / 低脂 / 规格)"] C2 --> C3["搜索筛选 / 属性标签 / 变体聚合"] end subgraph StoreSide["履约端：商品匹配 (Real-time)"] S1["订购商品图 + 现场拍摄图"] --> S2["多模态推理核对<br/>(如 AA 电池 vs AAA 电池)"] S2 --> S3["即时纠偏提示 (低延迟拦截)"] end ``` --- ### [06:08](https://www.youtube.com/watch?v=Quz16KujtZI&t=368s) 传统技术架构的局限与多模态 VLM 的引入 在生成式 AI 普及之前，传统技术方案在处理上述两个场景时均遇到了难以逾越的瓶颈： 1. **属性提取的历史方案**：最初为每个属性单独训练一个文本分类模型，但面对成百上千种属性时，模型的维护与运营成本呈指数级膨胀，根本无法扩展；后续尝试采用 OCR（光学字符识别）提取包装文本再清洗分析，但在实际落地中极其脆弱（brittle），鲁棒性极差。 2. **商品匹配的历史方案**：传统深度学习通常使用图像嵌入（Image Embeddings），将标准商品图与现场代购图映射到同一向量空间计算余弦相似度。但在充满背景杂物的强噪图像下，这种表征匹配迅速失效。此外，实际业务需要支持一定程度的“模糊匹配（Fuzzy Matching）”——例如顾客下单了 20 盎司的洗发水，在缺货时拿到 19 盎司同款往往优于直接缺货取消，而传统的硬性匹配难以兼顾这种业务弹性。 这两类场景共同具备四个核心维度：**多模态本质**、**高精度要求**（一旦误判代购商品会导致数百万美元的退款损失，属性错误则会破坏用户信任）、**输入强噪声**（来自数千家零售商参差不齐的数据源与极度模糊的现场拍照）以及**长尾杂货概念认知**（如 A2 牛奶、精瘦肉分类、特定自有品牌）。 多模态视觉语言模型（VLM）完美契合了上述四个维度的需求。单个 VLM 可以联合解析图像与文本，将业务规则灵活融入 Prompt 中执行模糊推理，并自然泛化到长尾概念。在属性提取时，输入图像与文本即可直接输出干净的属性值；在商品匹配时，结合现场杂乱照片与目录图片，VLM 能精准判断是否满足带业务规则的匹配逻辑。 --- ### [09:27](https://www.youtube.com/watch?v=Quz16KujtZI&t=567s) 生产落地挑战与自我反思（Reflection）优化机制 尽管 VLM 展现出强大的统一理解能力，但将其直接推向生产环境仍面临四大棘手挑战： - **幻觉问题（Hallucination）**：模型可能会极其笃定地将非素食产品标注为素食（Vegetarian），这对于特定饮食习惯的消费者而言是灾难性的体验。 - **高昂成本（Cost）**：对数千万商品提取数百个属性需要处理海量图像 Token，直接调用最先进的前沿闭源大模型会导致算力账单呈爆炸式增长。 - **决策随机性（Randomness）**：在输入、Prompt 及 Temperature 等推理参数完全一致的情况下，对于边界模糊的复杂样本，VLM 的判定结果仍有 6% 至 7% 的概率发生翻转。若代购员第一次拍照被系统拦截，用同一张照片重试却被放行，会产生极差的用户体验。 - **推理延迟（Latency）**：涉及多张高分辨率图像联合推理时，单次 VLM 调用可能耗时数秒，这对店内秒级响应的实时核验构成严峻挑战。 针对幻觉问题并进一步拉升模型精度，Instacart 引入了**自我反思（Self-Reflection）**机制。其原理类似于在人类做出判断后追问一句：“你确定这个答案完全正确吗？请再仔细核对一遍。”这种提示策略促使大模型进入批判性思维模式，重新审视图像与文本细节。在实际流水线中，系统首先让 VLM 执行第一轮属性提取，随后发起反思流程进行二次校验，实践证明该方法能显著提升最终输出的准确率。 ### [11:38](https://www.youtube.com/watch?v=Quz16KujtZI&t=698s) 双重思考反思与小模型微调：破解成本与延迟的双重瓶颈 在属性抽取流程中，为了最大程度规避模型幻觉，团队引入了一种被称为“双重思考”（double-thinking）的自我反思验证机制。当 VLM 从商品信息中抽取出一个具体属性值后，系统会发起二次调用，再次向 VLM 提问：“结合给定的属性规范指南，所抽取的数值是否正确？”只有当模型在二次确认中依然给出肯定答复时，该属性值才会被推送到下游的实际业务场景中使用。这一闭环极大提高了入库数据的准确性。 然而，直接在生产环境中依赖最顶尖的前沿 VLM（Frontier VLM）会面临严峻的工程挑战。前沿模型虽然准确率极高，但在构建实时交互体验时往往响应过慢，且调用成本高昂。为了兼顾延迟与成本，团队采用混合标注方案——结合人工标注与大语言模型标注构建高质量数据集，进而对参数量较小的轻量级 VLM 进行针对性微调（Fine-tuning）。这种方案不仅直接通过缩减模型体积带来了显著的吞吐与延迟收益，还带来了一个意料之外的巨大优势：对于长达数十页复杂提示词的模糊业务场景，小模型在微调过程中能够直接“内化并记住”杂货零售领域晦涩深奥的专业概念，无需在每次推理调用时都在 Prompt 中反复传递大段的规则定义。 ```mermaid flowchart TD accTitle: VLM 属性抽取与轻量化微调架构 subgraph S1["数据生成与反思验证"] A["多模态商品数据"] --> B["前沿 VLM 属性抽取"] B --> C{"自我反思验证\n'Double-thinking'"} C -- "确认有效" --> D["高质标注样本库\n(人工 + LLM 协同)"] C -- "未通过" --> B end subgraph S2["微调与生产部署"] D --> E["微调小参数量 VLM"] E --> F["精简版 Prompt\n(内化杂货领域专有知识)"] F --> G["低延迟、低成本下游实时服务"] end ``` - **双重思考校验**：通过闭环向模型反问验证结果是否符合指南规范，确保下游属性数据的纯净度。 - **微调的双重红利**：既大幅压降了模型自身的推理算力开销，又将长达数十页的 Prompt 大幅精简，实现延迟与 API 成本的双重缩减。 --- ### [13:03](https://www.youtube.com/watch?v=Quz16KujtZI&t=783s) 坚守“朴实无华”的基础建设：黄金数据集与 Agentic 自动化调优 演讲者分享了第三条看似“朴实无华”（unglamorous）却至关重要的核心经验：必须重金投入到黄金数据集（Golden Data Set）以及数据标注流程的建设中。在杂货电商的实际业务中，产品匹配等场景极具模糊性，系统面对的往往是充满噪点的包装图片，判断两款商品是否为同款绝非简单的“是/否”二元选择。商品在品牌、规格尺寸、口味、饮食偏好（如无麸质、低脂）等多个维度上错综复杂，甚至经常出现商品图片与文本描述相互矛盾的混乱情况。 在这种高度复杂且充满歧义的场景下，即便是通情达理的人类标注员之间也会产生严重分歧，更不用说人工操作本身还会产生疲劳和误标。如果缺乏高置信度的黄金数据集，算法团队的大量精力都会被浪费在排查标注规范漏洞或清洗脏数据上，而不是专注于模型与提示词调优。一旦建立了坚实的黄金测试集，不仅下游业务方能够真正信任评测指标（Eval Metrics），团队甚至可以构建完全无需人工介入的智能体调优闭环（Agentic Loop），让 AI 代理自动迭代优化 Prompt 并验证指标表现，从而省去海量的人力与时间成本。 - **多维度冲突挑战**：杂货数据涵盖品牌、规格、风味与饮食成分等繁杂维度，图文信息常有出入，极易引发人工误标与争议。 - **Agentic Loop 赋能**：依托高质量黄金基准集，系统可实现端到端的自动化 Prompt 优化与评估，免去人工反复介入。 --- ### [14:33](https://www.youtube.com/watch?v=Quz16KujtZI&t=873s) 问答环节：应对杂货库存波动的多源数据预测与代购员实时反馈 在问答环节，一位经营约 1,500 种 SKU 的零售业务从业者提出了关于库存阈值与缺货替代的实操困惑。提问者表示，在自身拥有全部库存的前提下，设定补货及在架阈值依然十分困难；而 Instacart 需要对接众多外部合作零售商的库存系统，面对店内库存仅剩 1 件这类边缘情况，系统究竟是选择不向用户承诺有货，还是先接单并寄希望于代购员（Shopper）到店后发起商品替换？ 针对该问题，演讲者解释道，Instacart 的底层逻辑要远比单纯依赖零售商库存接口复杂得多。由于实体杂货店内经常发生商品被顾客随手乱放（Misplaced）的情况，系统并不完全轻信零售商上报的静态数字，而是部署了专门的机器学习预测算法： ```mermaid flowchart LR accTitle: Instacart 智能库存判定流程 subgraph DataSources["多源输入"] A["历史周期性销售数据"] B["零售商实时库存接口"] C["一线代购员实时缺货上报\n(如连续多位确认)"] end subgraph Decision["智能判定与前置拦截"] D["库存聚合与 ML 缺货预测模型"] E{"判定为缺货?"} F["前台直接下架标记缺货\n(避免用户下单后落空)"] G["正常展示并开放加购"] end A --> D B --> D C --> D D --> E E -- "是" --> F E -- "否" --> G ``` - **周期性机器学习预测**：结合历史消费节律与商品流转周期，主动预测某款商品在特定时间段内耗尽的概率。 - **众包代购员实时动态感知**：充分利用穿梭在各门店的一线代购员作为动态传感器。如果上一位代购员反馈某货架已空，或者短时间内有两名代购员提交了相同的缺货汇报，系统聚合机制就会即时更新库存认知。 - **前置规避用户不满**：一旦模型判定商品实际已缺货，平台会主动在前台界面将其标记为不可购买，而不是让用户下单后再经历繁琐的替代与退款，以此保障消费体验。 --- ### [16:09](https://www.youtube.com/watch?v=Quz16KujtZI&t=969s) 问答环节：模型选型取舍与推理阶段的成本延迟权衡 随后，现场观众就开源与闭源模型的选型评估、模型部署服务框架（Model Serving）以及大规模推理落地时的成本优化展开了提问。观众指出，随着模型走向市场化实际应用，推理阶段的调用量将远远超过初期的训练阶段，企业该如何在性能、成本与开源闭源路线之间做出权衡。 演讲者指出，模型选型没有一刀切的标准，“一切高度取决于具体的业务任务”（"It's going to be highly task-specific"）。在 Instacart 的工程实践中，团队会综合权衡成本、延迟与准确率三个核心维度： - **延迟敏感型任务**：例如需要极速返回结果的实时搜索与交互场景，工程团队愿意在准确率上做适度的微小妥协，以换取极低的响应时间和小参数模型的高速推理。 - **低频长尾任务**：对于某些调用量极低、流量稀疏的离线或后台判定场景，成本和延迟都不是主要矛盾，团队可以直接调用业界最顶级的前沿模型（Frontier Model）来追求极致的准确性。 - **超大规模吞吐任务**：以核心的“商品属性抽取”（Attribute Extraction）为例，该场景需要扫描处理全平台数以亿计的庞大商品库，如果盲目采用前沿闭源大模型，产生的推理 API 开销将是“天文数字、令人无法承受”（"the cost is going to be outrageous"）。因此此类大流量核心场景必须坚定推行小模型微调与蒸馏落地方案。 ## Notable quotes > [00:04](https://www.youtube.com/watch?v=Quz16KujtZI&t=4s) 利用大模型跨越 2200 多万种独立商品进行结构化属性的提取、发现与自动修复。 — Pradeep Karuturi > [03:03](https://www.youtube.com/watch?v=Quz16KujtZI&t=183s) 许多核心健康卖点仅以微小图标或说明文字印在商品外包装上，必须通过商品包装图像才能获取，这使得属性提取本质上是一个多模态问题。 — Pradeep Karuturi > [06:08](https://www.youtube.com/watch?v=Quz16KujtZI&t=368s) 一旦误判代购商品会导致数百万美元的退款损失，属性错误则会破坏用户信任。 — Pradeep Karuturi > [09:27](https://www.youtube.com/watch?v=Quz16KujtZI&t=567s) 你确定这个答案完全正确吗？请再仔细核对一遍。 — Pradeep Karuturi > [11:38](https://www.youtube.com/watch?v=Quz16KujtZI&t=698s) 结合给定的属性规范指南，所抽取的数值是否正确？ — Pradeep Karuturi > [11:38](https://www.youtube.com/watch?v=Quz16KujtZI&t=698s) 小模型在微调过程中能够直接“内化并记住”杂货零售领域晦涩深奥的专业概念，无需在每次推理调用时都在 Prompt 中反复传递大段的规则定义。 — Pradeep Karuturi > [13:03](https://www.youtube.com/watch?v=Quz16KujtZI&t=783s) 如果缺乏高置信度的黄金数据集，算法团队的大量精力都会被浪费在排查标注规范漏洞或清洗脏数据上，而不是专注于模型与提示词调优。 — Pradeep Karuturi > [14:33](https://www.youtube.com/watch?v=Quz16KujtZI&t=873s) 一旦模型判定商品实际已缺货，平台会主动在前台界面将其标记为不可购买，而不是让用户下单后再经历繁琐的替代与退款。 — Pradeep Karuturi > [16:09](https://www.youtube.com/watch?v=Quz16KujtZI&t=969s) It's going to be highly task-specific — Pradeep Karuturi > [16:09](https://www.youtube.com/watch?v=Quz16KujtZI&t=969s) the cost is going to be outrageous — Pradeep Karuturi ## Mentioned Instacart, Pradeep Karuturi, Ai4, VLM, LLM, OCR, Agentic Loop, Frontier Model ## Video description Pradeep Karuturi - Machine Learning Engineer, Instacart In this talk, I will illustrate how Instacart leverages multimodal Large Language Models (VLMs) to transform grocery e-commerce by solving complex challenges in catalog intelligence and fulfillment. Traditional text-based or embedding-based models struggle with the noisy, real-world data inherent in grocery retail. I will explain how we successfully deployed VLMs to handle multimodal inputs, drastically improving product matching and attribute extraction across our massive catalog. You will also learn about our practical strategies for productionizing these models, including self-reflection techniques to mitigate hallucinations and knowledge distillation to optimize for cost and latency. [Source](https://www.youtube.com/watch?v=Quz16KujtZI)

Source quotes

[0:04] Pradeep Karuturi: 利用大模型跨越 2200 多万种独立商品进行结构化属性的提取、发现与自动修复。 [3:03] Pradeep Karuturi: 许多核心健康卖点仅以微小图标或说明文字印在商品外包装上，必须通过商品包装图像才能获取，这使得属性提取本质上是一个多模态问题。 [6:08] Pradeep Karuturi: 一旦误判代购商品会导致数百万美元的退款损失，属性错误则会破坏用户信任。 [9:27] Pradeep Karuturi: 你确定这个答案完全正确吗？请再仔细核对一遍。 [11:38] Pradeep Karuturi: 结合给定的属性规范指南，所抽取的数值是否正确？ [11:38] Pradeep Karuturi: 小模型在微调过程中能够直接“内化并记住”杂货零售领域晦涩深奥的专业概念，无需在每次推理调用时都在 Prompt 中反复传递大段的规则定义。 [13:03] Pradeep Karuturi: 如果缺乏高置信度的黄金数据集，算法团队的大量精力都会被浪费在排查标注规范漏洞或清洗脏数据上，而不是专注于模型与提示词调优。 [14:33] Pradeep Karuturi: 一旦模型判定商品实际已缺货，平台会主动在前台界面将其标记为不可购买，而不是让用户下单后再经历繁琐的替代与退款。 [16:09] Pradeep Karuturi: It's going to be highly task-specific [16:09] Pradeep Karuturi: the cost is going to be outrageous

Source quotes

[3:03] Pradeep Karuturi: the substrate for every shopping decision online [4:40] Pradeep Karuturi: classic product matching problem, but on steroids [6:08] Pradeep Karuturi: one model per attribute [11:38] Pradeep Karuturi: Hey, given the attribute guideline, is the extracted value correct? [14:33] Pradeep Karuturi: the cost is going to be outrageous

Original transcript

[0:04] Today I'm going to talk about a few generative  AI applications in grocery e-commerce. [0:10] This is the high-level agenda. First, I'll  give you a brief overview of Instacart, [0:15] and then we're going to look at a few  broad use cases of GenAI in grocery space. [0:21] And then I'm going to dive deep into a couple of  use cases in catalog and fulfillment intelligence. [0:27] And then for those use cases, we are going to  look at challenges both pre and post GenAI era. [0:33] And then I'll conclude the talk  with some takeaways and learnings. [0:39] So, for some context about Instacart, we are the  leading grocery e-commerce platform in the US, [0:45] reaching more than 98% of American households.  And based on the last quarter results, [0:51] we have more than $40 billion in  annualized GTV. And over the last 1 year, [0:58] more than 26 million customers have placed  over 350 million orders across 2,200 retailers. [1:06] We give the customers the choice to shop  from nearly 100,000 stores while creating [1:12] earnings flexibility opportunities for shoppers  who actually go in stores to fulfill those orders. [1:20] Now, with that, let's look at a few  use cases. This list is by no means [1:26] exhaustive. It's just a very broad overview  of some of our use cases, starting with the [1:32] absolute basic for any e-commerce platform. We use  GenAI solutions to enrich our catalog, to extract, [1:41] discover, and heal structured attributes across  more than 22 million unique items. We also have [1:47] use cases in search and personalization, for  example, in query understanding and query [1:52] rewrites. And in terms of personalization,  in uh understanding and recommendations. [1:58] So, at the bottom of the slide, I've linked  to a couple of papers that my colleagues [2:02] have published at SIGIR and KDD this year. Uh we  won't get into detail into those, but one of them [2:08] talks about how we leverage LLMs to merchandise  our storefronts in a more personalized manner. [2:14] And the other talks about how we leverage LLMs  to generate user intents using search queries. [2:21] We also have a conversational shopping assistant  that helps our users parse the recipes and [2:25] build their baskets and navigate our catalog  according to their preferences. And in the end, [2:32] once you place an order, our shoppers go in stores  to fulfill those orders. And uh in that process, [2:38] there are a few avenues where things could go  wrong, and uh we're going to dive deep into one of [2:42] those use cases. So, basically, I'm going to zoom  into either ends of this slide, how we leverage [2:48] multimodal LLMs or VLMs, uh in catalog and  fulfillment intelligence applications. So, for the [2:55] rest of the talk, whenever you see a VLM or visual  language model, just treat it as a multimodal LLM. [3:03] So, uh when customers are shopping for groceries  online, they're looking for various kinds of [3:09] information. Is this product organic? Is this  product What is the sugar content in that in [3:14] that product? Or what is the size and things  like that? All those questions are answered by [3:19] structured catalog attributes. Uh Now, if we  have missing attributes or wrong attributes, [3:26] it could mean that the customer  may not find what he's looking for, [3:29] or they may find the wrong attributes. So,  really, catalog is the substrate for every [3:34] uh shopping decision online. So, if you look at  this uh image on the screen, uh we can see that [3:42] this product is uh good source of fiber and that  it is gluten-free, and that it is low in fat. But, [3:48] we don't get this information from the text that  the retailer send us. You have to actually look at [3:53] the product images to get this information. So, in  that sense, this problem is inherently multimodal. [4:00] Now, all these attributes form the foundation for  multiple experiences on Instacart. Uh for example, [4:07] we show attribute badges for some uh products.  For example, this product is gluten-free. [4:13] And uh we also use these attributes to power  our variance experience. We have clustered like [4:18] different flavors of chips uh in one item space.  And uh we also use these attributes uh to help [4:24] our users narrow down their search results.  And we have many more applications in health [4:30] so that our customers can make the right  choices. Now, our catalog has a pipeline [4:34] to extract all these different kinds of  attributes uh with very high precision. [4:40] Now, uh if you go to the other end of the  funnel, uh once a customer place an order, [4:45] a shopper actually goes in store to  pick those items and fulfill the order. [4:49] And usually, we validate the shopper  picks using barcode. But sometimes, [4:54] for a wide variety of reasons, uh we can't  just use uh barcode to validate the items. [4:59] So, we essentially need a different mechanism to  validate what the shoppers are picking in store. [5:06] Uh so, we use the shopper photos uh from  store to validate that. So, on the screen, [5:12] I've included an image uh where a customer has  ordered a double-A battery pack. And the shopper [5:17] in store is actually trying to fulfill it with a  triple-A battery pack. It may not look like that. [5:22] Uh it's it could be a genuine human oversight.  And we want to leverage that image information to [5:27] nudge the shopper to do the right thing there. Uh  so, this is the classic product matching problem, [5:33] but on steroids. Uh The reason I say  that is we deal with really noisy images. [5:39] And uh I've included a couple of examples of  shopper images, like all those background noise. [5:45] And believe me, these are some of the better ones.  We see. Way worse uh shopper photos, and we need [5:49] to reason about these images. When they are blurry  and weird angles and things like that. So, one [5:54] thing to note about this application is uh this  is inherently real-time application as opposed [6:01] to offline processes for catalog. So, latency  is going to be an issue for this application. [6:08] Now, for both these applications, attribute  extraction and uh product matching, [6:13] what has been hard historically  speaking? For attribute extraction, we uh [6:19] traditionally relied on text-based models, one  model per attribute, uh but uh that is very hard [6:25] to scale to hundreds of different attributes  because because of the operational challenges. [6:30] And moreover, as we have seen, most of the  attribute information is in product images. So, [6:35] how would you deal with that?  You could say, "Hey, you can use [6:38] OCR, optical character recognition, extract the  text, clean the text, and extract the attributes." [6:43] But, in reality, that turned out to  be very brittle and doesn't scale. [6:48] Uh now, for product matching approaches, we  also tried the traditional deep learning route [6:54] of using image embeddings, where we uh embed the  in-store shopper images and uh customer ordered [7:00] product images in the same space and look for  image similarity. But, in reality, that approach [7:07] fell apart pretty quick, especially with the noisy  images that we have seen in the previous slide. [7:13] And one nuance to add here is that our product  folks uh want to have some flexibility in product [7:18] matching. This is not an exact match problem. It's  sort of like a fuzzy match fuzzy match problem. [7:24] Uh for example, if a customer has  ordered a 20-oz shampoo bottle, [7:28] maybe he'll be okay with getting a 19-oz shampoo  bottle rather than not getting anything at all. [7:36] So, some common threads uh across both these use  cases is both of them are inherently multimodal. [7:43] Uh you have to look at both images and text to  actually make a decision. And precision is really [7:49] critical because if you make the wrong decision  on a shopper pick, that could lead to refund costs [7:55] running at million millions of dollars for  Instacart. And if there are wrong attributes, [7:59] that would break customer trust in the platform.  And we have seen that we deal with really noisy [8:04] inputs both from catalog data perspective because  we are dealing with thousands of retailers with [8:10] varying catalog data quality, but also we have  seen how noisy the shopper photos from store are. [8:17] The another grocery related nuances we need to  understand long-tail grocery concepts. Like what [8:22] is an A2 milk? What is a lean meat or private  label store brands because all these nuanced [8:27] attributes are important when you're doing product  matching or actually extracting these attributes. [8:33] Now any solution we come up with needs  to hit all four of these dimensions. [8:38] And the This is uh really where VLMs or visual  language models uh really came to our rescue. [8:44] Uh one VLM model reads both text and image jointly  and can reason about uh both those inputs along [8:52] with any business rules that you may have. And it  generalizes to the long-tail concepts as well. Now [8:59] in the case of attribute extraction, you pass in  the image of the product and the text and ask it, [9:04] "Hey, is it product gluten-free?" And you will get  a clean type answer. And in the case of product [9:09] matching, you pass the store in-store noisy photo  and then the product image from the catalog and [9:15] the product text and ask for the product matching  decision. So one thing to note here with VLMs is [9:22] they're capable of doing fuzzy matching because I  can incorporate business rules into the prompt. So [9:27] that that's a win as well. However, not everything  is rosy. Uh VLMs come with their own challenges. [9:35] The first problem is hallucination. A VLM  can confidently label a non-vegetarian [9:40] product as vegetarian and that could be  a disaster for a vegetarian customer. [9:45] The second is cost. When you're extracting  hundreds of attributes across millions of products [9:51] and especially when you're doing multimodal  inference because images consume a lot of tokens. [9:55] Uh the your cost can balloon into an expensive  bill. So naively routing to the most expensive [10:01] frontier model is not going to scale. The third  interesting problem is actually VLM randomness. [10:07] For the exact same inputs, same temperature,  other parameters, all everything being the same, [10:13] we have noticed that especially  for our more ambiguous use cases, [10:17] the decision of the VLM can change 6 to 7%  of the times. That creates challenges in [10:23] building user experiences because let's assume we  blocked a shopper from picking a particular item [10:29] on the first try and the second time he  still retries with the same photos and [10:35] somehow we let it through. That creates a jarring  user experience because of the randomness. The [10:42] fourth thing is latency, especially for real-time  use cases, latency is going to be a concern, [10:47] especially since we're using multiple images  which could take multiple seconds in the VLM call. [10:55] So however, not all is a gloom. I'm going to talk  about a few of the techniques that we have used to [11:03] solve all the challenges that I outlined.  So to tackle the problem of hallucination [11:09] and improve the model precision for our task, uh  you can use a simple reflection. It's basically [11:15] asking the model, "Hey, can you double check  your answer?" It's very much like asking humans, [11:21] "Hey, are you really sure about your answer?" and  it puts them into a more critical thinking mode. [11:26] And it turns out the precision of the model can  increase with this technique. Uh it's outlined in [11:31] the image uh, on the right side. Basically, we  first ask the VLM to extract uh, an attribute. [11:38] And once we get an extracted  value, you again ask the VLM, [11:42] "Hey, given the attribute guideline, is the  extracted value correct?" It's the double-thinking [11:46] approach. And if it still says yes, only then uh,  we push that attribute to downstream use cases. [11:54] Uh, now let's talk about LLM fine-tuning to  solve the twin problems of cost and latency. [12:01] A frontier VLM can be very accurate,  uh, but in practice, it can be very slow [12:07] in when building real-time experiences. So, uh,  without going into too much detail, we uh, create [12:14] high-quality data sets using a mix of human and  uh, LLM labelers, and we fine-tune a small VLM. [12:21] So, you're essentially getting latency and cost  wins because the fine-tuned VLM is a much smaller. [12:28] Another interesting thing to note here  is, especially for ambiguous use cases, [12:32] when your prompt or instructions  are running into tens of pages, [12:37] in fine-tuning process, the smaller VLM can  actually learn esoteric grocery concepts without [12:44] you having to pass these definitions in  every single prompt call. So, in a sense, [12:50] the prompt itself becomes simpler. So,  you're getting both latency and cost wins, [12:57] not just from the model being smaller,  but also your prompt becoming simpler. [13:03] So, the third unglamorous lesson is invest in your  golden data sets and your data labeling processes. [13:11] Uh, because especially in uh, ambiguous use  cases, uh, for example, in product matching, [13:17] we deal with really noisy images where it's not  always clear is it a yes or a no decision. We [13:24] are dealing with multiple different dimensions in  brand, size, flavor, dietary and other attributes. [13:31] And sometimes to make matters more  complicated, the information and image and text [13:36] conflict with each other. So, what do you do? Even  reasonable humans can disagree with the labels. [13:42] And on top of that, humans can also commit  mistakes and actually mislabel things. [13:47] So, if you don't have a reliable golden data set,  you're actually spending more time debugging your [13:51] guidelines or data set than actually doing proper  prompt tuning. And once you have a reliable golden [13:58] data set, you can your downstream customers can  actually trust your eval metrics. The other thing [14:04] is once you have a reliable golden data set, you  can set an agentic loop to actually prompt tune [14:09] and improve your metrics without having any human  in the loop. You're saving a ton of time cost. [14:17] So, with that I'll end my talk. [14:21] Uh if there any questions, I'm happy  to take them. Yeah, a few minutes. [14:32] [applause] [14:33] So, I own a retail operation with about 1,500  SKUs, which is like nothing compared to what [14:40] you're doing here. And it's my own inventory, and  I struggle with like what thresholds to set things [14:45] at with my inventory to ensure that it's going to  be in stock. And I know you guys do substitutions, [14:50] and you have to count on a lot of different  retailers and stores having their inventory [14:55] correct. How do How do you guys handle  that? Is there a certain threshold where [15:00] if something is like one inside of a store, then  you're not going to like make that promise to [15:04] the customer that exists, or you're going to  go ahead and take that order, and then just [15:08] move to the substitution substitution phase  when the shopper gets there? Uh we actually [15:13] do something much more complicated because we  don't entirely on the inventory information, [15:19] and sometimes in grocery stores and things  may be misplaced, too. We actually have a uh [15:25] ML algorithms using historical cyclical data, we  actually predict when things may be out of stock, [15:34] and then we have continuous stream of information  from the shoppers who are actually going into the [15:38] store, "Hey, the previous order this was  in stock, that was This was out of stock." [15:43] And let's say we get information from two  different shoppers saying the same thing. We [15:47] have mechanisms to aggregate information and have  a a bit more intelligent inventory information. [15:53] So, uh I don't know if I answered your question.  So, and if we think some product is out of stock, [15:59] we actually uh uh mark it as out of stock so  that the customers won't be able to shop for it, [16:05] so that there won't be customer dissatisfaction  later on. I see. I see. I see. And is [16:09] that's based off of like a certain time period  where is you've gotten feedback the last like day [16:15] or something that it's out? How do you choose the  models for you to use, and how you do the model [16:22] serving and harness, and that's going to give you  the whole system? Especially in light of a lot [16:27] of conversations that's been happening recently  between close models, open models, performance, [16:32] as well as cost optimization, especially as you  actually take it to the market and its actual and [16:39] inference is going to be a lot more volume than  initial training. So, wondering how you actually [16:47] make those choices, and if you see any shifts  um I either now or in the distant future. Uh [16:55] It's going to be highly task-specific, that's  my generic answer. But, we're going to look at [16:59] multiple different dimensions, the cost aspect,  latency aspect, uh For example, in in my talk, [17:07] some use cases are very latency sensitive, so  we can trade off on the accuracy a little bit [17:13] while getting the model to answer faster.  And then, there may be some use cases where [17:18] neither cost or latency may be an issue because  the traffic is so low for that use case. [17:24] We can put the most absolute frontier model  there. Uh but for our attribute extraction, [17:31] we can't put the frontier model there  because the cost is going to be outrageous.