Staples Replaces Pure Vector Search With Multi-Stage AI Pipeline

Solving Product Matching with AI: Query Understanding, Hybrid Retrieval, and Multi-Stage Ranking — Ai4

Ai4

Vector embeddings inherently fail on exact specifications like yields and packaging units, requiring enterprise search to handle bulk candidate retrieval with mathematical models and reserve generative AI strictly for final attribute adjudication.

AI-assisted summary of Solving Product Matching with AI: Query Understanding, Hybrid Retrieval, and Multi-Stage Ranking — original source. Check the original for full context.

Source published: 2026-08-14

Detailed notes

## Background Published on YouTube by Ai4 on August 14, 2026, this presentation features Sidharth Thapar, AI Manager and Senior Applied Scientist at Staples Inc. The talk details how enterprise search systems achieve exact product matching beyond basic vector similarity by combining query understanding, hybrid retrieval, machine-learned ranking, LLM reasoning, and human-in-the-loop workflows.

## TL;DR Enterprise product search often fails when standard semantic models conflate similar variants—like confusing standard-yield with high-yield toner cartridges or misinterpreting units of measure. To solve this zero-tolerance challenge, Thapar outlines a four-stage architecture that normalizes queries deterministically with selective LLM rewrites, retrieves broad candidates via hybrid BM25 and dense vector fusion with multimodal gating, re-ranks via cross-encoders, and uses LLM reasoning on top candidates to verify attribute equivalence. The pipeline balances high recall and compute efficiency by retrieving with math and adjudicating with language, supported by account personalization and human-in-the-loop active learning.

## Atmosphere & style The presentation carries a pragmatic, engineering-grounded tone, blending deep technical architecture with relatable humor about standing between the audience and their lunch. Thapar speaks with the seasoned authority of an industry practitioner tackling unglamorous yet critical pain points like cryptic abbreviations, unit mismatches, and inference cost constraints. The talk delivers a disciplined, methodical progression from real-world failure modes to robust multi-stage system designs, making complex enterprise trade-offs feel tangible and structured.

## Key takeaways - [00:04](https://www.youtube.com/watch?v=IzmgXhrAwcM&t=4s) Traditional vector similarity searches optimize for semantic resemblance rather than exact equivalence, creating severe blind spots for part numbers, yield capacities, and units of measure in enterprise procurement. - [01:15](https://www.youtube.com/watch?v=IzmgXhrAwcM&t=75s) Enterprise product matching must satisfy diverse organizational trade-offs across e-commerce search, offline contract quoting, pricing intelligence, and catalog onboarding using a single configurable pipeline. - [04:40](https://www.youtube.com/watch?v=IzmgXhrAwcM&t=280s) Search engineering teams must map business KPIs to distinct machine learning metrics: Recall@K for catalog retrieval completeness, NDCG for ranking position, and explicit overturn rates for error tolerance. - [06:22](https://www.youtube.com/watch?v=IzmgXhrAwcM&t=382s) A multi-stage architecture balances speed and precision by adhering to the core principle: retrieve early for high recall, then rank late for high precision. - [07:57](https://www.youtube.com/watch?v=IzmgXhrAwcM&t=477s) Query understanding uses a two-step approach—deterministic rule extraction for common patterns and selective LLM rewriting for ambiguous long-tail queries—to control inference costs. - [09:35](https://www.youtube.com/watch?v=IzmgXhrAwcM&t=575s) Hybrid candidate retrieval combines lexical BM25 with dense semantic vectors via Reciprocal Rank Fusion (RRF) to capture exact alphanumeric part numbers while preserving synonym coverage. - [11:25](https://www.youtube.com/watch?v=IzmgXhrAwcM&t=685s) Multimodal retrieval and late-interaction vision models (like ColPali) should be confidence-gated, running only when text data is too sparse to distinguish visually distinct products. - [13:01](https://www.youtube.com/watch?v=IzmgXhrAwcM&t=781s) Re-ranking models follow an interaction ladder from fast bi-encoders up to computationally heavy cross-encoders with early token interaction for sharp discriminator scoring. - [15:04](https://www.youtube.com/watch?v=IzmgXhrAwcM&t=904s) A final LLM reasoning stage conducts attribute-by-attribute audits on the top 3–5 candidates to distinguish genuine functional equivalence from mere contextual resemblance. - [16:50](https://www.youtube.com/watch?v=IzmgXhrAwcM&t=1010s) Personalization and multi-objective ranking balance user purchasing history with corporate margin targets and catalog diversity, evaluated through margin-weighted holdout cohorts. - [18:05](https://www.youtube.com/watch?v=IzmgXhrAwcM&t=1085s) Enterprise search architectures should follow the foundational guideline: 'retrieve with math, adjudicate with language' to keep latency and token costs within strict production SLAs. - [19:27](https://www.youtube.com/watch?v=IzmgXhrAwcM&t=1167s) Human-in-the-loop workflows route low-confidence adjudication pairs to domain experts, generating hard negatives for continuous active learning and laying the groundwork for autonomous agentic commerce.

## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=IzmgXhrAwcM&t=4s) The Yield Gap and the Lunchtime Dilemma

A business customer types a seemingly straightforward query into an enterprise search bar: `HP 26X black TNR CTG 2-pack`. To an everyday search engine, returning an HP black toner cartridge might look like a triumphant match. But beneath the surface lies a critical discrepancy: the requested cartridge is an HP 26X, a high-yield model designed to print roughly 9,000 pages. If the search engine instead returns a standard-yield HP 26A that only produces 3,000 pages, the customer receives a third of what they expected. This precise gap—where a system returns a product that looks nearly identical but fails the test of exact equivalence—is the crux of the enterprise matching problem.

Sidharth Thapar, an AI manager and senior applied scientist at Staples, frames this challenge not as a failure of modern machine learning, but as an inherent property of how these models are built. As he jokes to the audience, acknowledging that he is "the only one standing between you and your lunch" for the next twenty minutes, traditional semantic models are trained specifically to identify *similar* items, not necessarily the *exact same* item. In consumer search, close enough is often good enough; in enterprise procurement, a near miss is an expensive failure.

- The motivating query `HP 26X black TNR CTG 2-pack` highlights the risk of confusing standard yield (3,000 pages) with high yield (9,000 pages). - Vector similarity optimizes for semantic proximity rather than exact identity, creating a systematic blind spot for part numbers and yields.

---

[01:15](https://www.youtube.com/watch?v=IzmgXhrAwcM&t=75s) Why Product Matching Fails in the Enterprise

Product matching across enterprise systems contends with catalogs spanning millions of items, where inbound product data is notoriously sparse, inconsistently formatted, and crammed with cryptic abbreviations like `PK` for pack and `TNR` for toner. Unlike general web search where a user browses a wide list of plausible links, enterprise workflows typically require identifying the single correct item out of dozens of superficially identical catalog variants. This comes with a zero-percent tolerance for error: shipping the wrong consumable triggers costly customer returns, logistical waste, and broken customer trust.

```mermaid flowchart TD accTitle: Enterprise Use Cases for Product Matching subgraph UC["Enterprise Workflows"] S["Search • Latency: Tens of ms • Tolerance: Tolerates pack mismatches"] Q["Quoting • Latency: Offline batch • Tolerance: Near-perfect matches required"] P["Pricing Intelligence • Latency: Moderate • Tolerance: Must be explainable"] C["Catalog Onboarding • Latency: Flexible • Tolerance: High coverage, lower precision"] end ```

Rather than building siloed, bespoke algorithms for every department, enterprise architects must address a matrix of competing trade-offs across distinct business functions:

- **E-Commerce Search:** Demands strict sub-second response times in the tens of milliseconds. While returning a 24-pack instead of a 12-pack is suboptimal, users can manually filter, making low latency the primary constraint. - **Quoting Systems:** Generate complex contract bids offline in batch mode. Latency is virtually irrelevant, but matching accuracy must be near-perfect to avoid underbidding or quoting unavailable stock. - **Pricing Intelligence:** Scrapes and matches competitive offerings to establish margin strategies. The results cannot rely on black-box vector scores; pricing analysts need fully explainable feature-by-feature parity before adjusting price points. - **Catalog Onboarding:** Ingests vast external distributor feeds. High catalog coverage is the primary mandate—identifying candidate matches for everything being imported—even if it temporarily introduces lower precision downstream.

The core engineering objective is therefore not to build fragmented point solutions, but to design a single, flexible pipeline with customizable levers that can be tailored across these contrasting organizational demands.

---

[03:35](https://www.youtube.com/watch?v=IzmgXhrAwcM&t=215s) The Nuance of Exactness: Units, Policies, and Scale

The difficulty of enterprise matching stems from nuanced business rules that pure semantic models consistently violate. A model trained on embeddings might treat a 12-pack and a 24-pack as practically synonymous because their textual descriptions share 95% of their vocabulary. Yet in commerce, substituting a 12-pack for a 24-pack can double the price or short-change an entire office floor. Similarly, hardcoded corporate policies dictate brand relationships: if a client explicitly searches for a national brand like an HP LaserJet printer, the algorithm must not substitute a private-label alternative, even if the feature sets match completely.

Units of measure (UOM) introduce another minefield. Searching for a "12-box pack" versus a "12 box" represents a twelve-fold pricing multiplier that vector search cannot reliably arbitrate. While engineering teams often resort to rule-based post-filtering, there is a hard limit to how many brittle regex patterns and hardcoded overrides an enterprise can maintain. Simultaneously, high-precision ranking architectures like cross-encoders offer the early-interaction attention mechanisms needed to resolve fine distinctions, but they are computationally expensive and far too slow to evaluate raw candidate pools across millions of records.

- Semantic models frequently confound "similar cousins" (e.g., 12-pack vs. 24-pack) due to high textual overlap. - Hardcoded business policies (such as preventing private-label substitutions for Tier-1 national brands) must be respected without bloating rule engines. - Unit-of-measure mismatches can alter pricing by orders of magnitude (e.g., a single box vs. a case of twelve boxes). - Cross-encoders provide deep feature scrutiny but fail to meet latency budgets when exposed to millions of candidate documents.

---

[04:40](https://www.youtube.com/watch?v=IzmgXhrAwcM&t=280s) Metric Alignment: Mapping Business KPIs to Machine Learning

Before assembling model architectures, engineering teams must anchor their evaluation frameworks to concrete business KPIs rather than generic machine learning metrics. A disconnect between operational goals and statistical loss functions inevitably produces search pipelines that look great on paper while failing in production.

```mermaid flowchart LR accTitle: Evaluation Metric Alignment Pipeline subgraph Business["Business Questions"] B1["Can we find the product at all?"] B2["Is the right product at the top?"] B3["How often are we catastrophically wrong?"] end

subgraph Metrics["ML Metrics"] M1["Recall@100 (Target: ~98%)"] M2["NDCG@10 / NDCG@1"] M3["Overturn / Error Rate (Zero-Tolerance Budget)"] end

B1 --> M1 B2 --> M2 B3 --> M3 ```

The evaluation strategy breaks down into three distinct checkpoints:

1. **Retrieval Completeness:** Answering the operational question "Can we find the product at all?" requires prioritizing Recall@K (such as Recall@100). For an enterprise catalog, the system should consistently capture the correct target in roughly 98% of queries within its candidate pool. 2. **Rank Positioning:** Simply surfacing a product somewhere in the top hundred items does not translate into revenue or quoting speed. Teams must track Normalized Discounted Cumulative Gain (NDCG), measuring NDCG@10 for general browsing or NDCG@1 when an operational system requires the absolute single best match at the top position. 3. **Error Tolerance:** Answering "How often are we wrong?" addresses the zero-tolerance constraint. Teams must establish an explicit budget for the overturn rate—determining whether the business can absorb a 5% margin of error, or whether high-risk use cases require aggressive vetoing mechanisms to prevent false-positive deliveries.

---

[06:22](https://www.youtube.com/watch?v=IzmgXhrAwcM&t=382s) Architecture: The Multi-Stage Matching Pipeline

To balance scale, speed, and exactness, Thapar proposes a four-stage hybrid architecture. The fundamental philosophy governing this pipeline is simple: **retrieve early for high recall, then rank late for high precision**. Because high-accuracy models are too computationally expensive to run over an entire multi-million-item database, the system progressively winnows the candidate set through increasingly sophisticated, compute-intensive evaluation stages.

```mermaid flowchart LR accTitle: Multi-Stage Product Matching Pipeline subgraph S1["Stage 1: Normalize"] Q["Raw Input (Text/Image/Doc)"] --> QU["Query Understanding • Regex / Dictionaries • Selective LLM Rewrite"] end

subgraph S2["Stage 2: Retrieve"] QU --> RET["Hybrid Retrieval • BM25 (Lexical) • Dense Vectors • ColPali (Multimodal) • Reciprocal Rank Fusion"] end

subgraph S3["Stage 3: Rank"] RET --> RK["ML Re-Ranking • Cross-Encoder • Interaction Ladder"] end

subgraph S4["Stage 4: Reason"] RK --> LLM["LLM Reasoning • Attribute Verification • Equivalence vs Resemblance"] end

LLM --> Out["Final Match"] ```

The pipeline begins with query understanding, which Thapar candidly calls "the most least cool part of the system, but this is the one that actually still solves the real pain in the system that comes later." Unstructured, messy input—whether a raw text string, an invoice scan, or an image—is normalized into a canonical, machine-readable query representation. Once standardized, the candidate generation phase utilizes hybrid retrieval to extract the top several hundred plausible candidates from the catalog, maximizing recall.

From this narrowed candidate pool, machine-learned re-ranking models apply early-interaction attention to score structural nuances. Finally, the top shortlist passes to an LLM reasoning stage. Rather than relying on scalar similarity scores, the reasoning engine performs an attribute-by-attribute audit, validating functional equivalence before confirming a match.

---

[07:57](https://www.youtube.com/watch?v=IzmgXhrAwcM&t=477s) Stage 1: Two-Step Query Understanding

Messy enterprise queries often arrive in shorthand, such as `HP 2X 26X BLK TNR CTG 2-pack`. The query understanding layer is tasked with translating this cryptic string into a fully enriched canonical representation: `HP LaserJet toner cartridge high yield 26X black 2-pack`. This transformation operates as a deliberate two-step mechanism designed to keep operational costs low while maintaining high fidelity across the long tail of search traffic.

The first step relies on deterministic attribute extraction using controlled dictionaries, regular expressions, and fuzzy matching. It extracts entities such as brand (`HP`), line (`LaserJet`), cartridge yield code (`26X`), color variations (`BLK`, `BLACK`, `BLK.` mapped systematically to canonical `black`), and packaging configurations. While deterministic systems handle the head and torso of catalog traffic efficiently, they break down when faced with novel, unseen abbreviations or chaotic distributor syntax.

```mermaid flowchart TD accTitle: Two-Step Query Understanding Process In["Raw Query: HP 26X BLK TNR CTG 2-pack"] --> D["Step 1: Deterministic Extraction • Controlled Dictionaries • Regex & Fuzzy Matching"] D --> Check{"Unseen / Ambiguous Tail?"} Check -- No --> Canon["Canonical Query: HP LaserJet toner cartridge high yield 26X black 2-pack"] Check -- Yes --> LLMR["Step 2: LLM Rewrite • Invoked only for long tail • Standardizes unknown syntax"] LLMR --> Canon ```

To bridge this gap without running up massive inference bills, the system employs selective LLM rewriting. The overarching design principle is clear: "We use LLMs only where it's needed to optimize our cost while getting the best results here." The LLM is invoked only for the ambiguous, long-tail queries that fail initial entity detection, converting irregular terms into the controlled vocabulary required by downstream retrieval systems.

- Deterministic normalization uses dictionaries and fuzzy regex to map messy variants (e.g., `BLK`, `BLK.`) to canonical terms (`black`). - Selective LLM invocation protects margins by reserving expensive generative calls strictly for the ambiguous, unmapped long tail. - Controlled vocabularies prevent downstream vector indexes from degrading due to out-of-vocabulary terms.

---

[09:35](https://www.youtube.com/watch?v=IzmgXhrAwcM&t=575s) Stage 2: Hybrid Candidate Retrieval and Fusion

Candidate retrieval over pure text traditionally pits lexical matching against dense semantic retrieval. Lexical algorithms like BM25 excel at preserving exact token matches—such as isolating the model code `26X` within part number `CF226X`. A standard dense vector embedding model frequently overlooks or blurs single-character token differences, causing part numbers to blend together. Conversely, BM25 fails completely when domain terminology shifts—failing to recognize that a "print cartridge" and a "toner cartridge" refer to the identical piece of office hardware.

Dense semantic vectors, especially when fine-tuned on proprietary enterprise transaction logs, readily bridge domain synonyms. However, they introduce a hazardous blind spot: returning a standard-yield `CF226A` cartridge when the user explicitly queried the high-yield `CF226X`.

Because neither paradigm succeeds in isolation, the architecture combines them via Reciprocal Rank Fusion (RRF). RRF blends the disparate candidate lists based on relative rank positions rather than attempting to calibrate unnormalized raw similarity scores. This hybrid retrieval guarantees that both standard-yield (`CF226A`) and high-yield (`CF226X`) models enter the candidate set, achieving the high overall recall required before moving to precision scoring.

---

[11:25](https://www.youtube.com/watch?v=IzmgXhrAwcM&t=685s) Multimodal Retrieval and Vision Confidence Gating

Enterprise inputs are not always cleanly typed text strings; they often arrive as sparse descriptions, noisy PDF specification sheets, or unstructured catalog images. When product text is exceptionally thin—such as a bare catalog entry reading `HP CTG BLK`—textual search cannot distinguish between a black toner cartridge and a printer drum unit. Both components belong to the same printer ecosystem, share identical generic titles, and use identical color labels, yet represent entirely different physical mechanisms.

```mermaid flowchart TD accTitle: Vision Confidence Gating Mechanism Input["Input: HP CTG BLK (Toner vs Drum Unit)"] --> TextConf{"Is text sufficiently rich & confident?"} TextConf -- Yes --> DenseText["Standard Hybrid Text Retrieval"] TextConf -- No / Low Confidence --> VisionGate["Multimodal Confidence Gating"] VisionGate --> DualEnc["Dual Encoders / ColPali • Late Interaction on Tables/Specs • Visual Differentiation"] DenseText --> Merge["Candidate List"] DualEnc --> Merge ```

To resolve these ambiguous edge cases, the pipeline integrates multimodal retrieval. Standard dual-encoder vision-language models can separate drums from cartridges by examining overall form factors, but they often struggle to read minute alphanumeric typography printed on product packaging (such as spotting the tiny difference between `26A` and `26X`). For complex visual inputs like spec sheets and spec tables, modern late-interaction architectures like ColPali apply token-level patch comparisons to capture both tabular layout and fine print.

Because ColPali and related visual late-interaction models are computationally demanding, Thapar advocates for **confidence gating**: - Computer vision and multimodal models are held in reserve rather than evaluated on every query. - The system assesses the confidence and richness of the textual query first. - Only when textual confidence drops below a set threshold does the gateway trigger visual embedding extraction, preventing unnecessary inference latency.

---

[13:01](https://www.youtube.com/watch?v=IzmgXhrAwcM&t=781s) Stage 3: The Interaction Ladder in Machine-Learned Re-Ranking

Once hybrid retrieval isolates a few hundred candidate products, Stage 3 introduces machine-learned re-ranking to sort them with high precision. Re-ranking models can be trained under pointwise, pairwise, or listwise loss objectives to optimize the top-of-page ordering. The key architectural decision centers on where and how the query tokens interact with the candidate document tokens—a progression Thapar illustrates through the "interaction ladder."

```mermaid flowchart TD accTitle: The Interaction Ladder of Retrieval and Ranking subgraph Ladder["The Interaction Ladder"] direction TB BE["Bi-Encoder (No Interaction) • Independent embeddings • Ultra-fast / Low latency • Coarse accuracy"] LI["Late Interaction (e.g., ColPali / ColBERT) • Token-level MaxSim • Moderate speed • Preserves fine-grained layout/specs"] CE["Cross-Encoder (Early Interaction) • Joint all-to-all attention • High compute / High latency • Sharpest precision"] end BE --> LI --> CE ```

At the base of the ladder, bi-encoders compute embeddings for query and document independently, comparing them via a fast dot product without token-to-token interaction. Higher up the ladder, late-interaction frameworks (such as ColBERT or ColPali) preserve token-level embeddings and compute maximal similarity scores (`MaxSim`), capturing richer sub-phrase alignment at moderate latency. At the top of the ladder sit cross-encoders featuring full early interaction: query and document tokens are concatenated and fed jointly through every self-attention layer.

While cross-encoders produce the sharpest discriminative scoring, their quadratic attention cost confines them strictly to the top candidate pool. Even with this power, however, scalar-scoring cross-encoders still harbor fatal blind spots. A cross-encoder evaluating a query for a `12 box` against a catalog item for a `12-pack` might assign a sky-high similarity score of 0.96 due to token overlap, failing to realize that the commercial units represent entirely incompatible packaging hierarchies.

- Cross-encoders provide deep early token interaction, identifying near-identical product variations at the expense of computational throughput. - Computational overhead limits cross-encoder deployment strictly to candidate shortlists rather than full catalogs. - High vector similarity scores (e.g., 0.96) can still produce false positives by glossing over non-negotiable unit-of-measure boundaries.

---

[15:04](https://www.youtube.com/watch?v=IzmgXhrAwcM&t=904s) Stage 4: LLM Reasoning for Attribute Equivalence

To eliminate the stubborn false positives that survive Stage 3 re-ranking, the pipeline introduces a dedicated LLM reasoning layer. While machine-learned re-rankers excel at scoring textual and contextual *resemblance*, they do not truly *reason* through functional equivalence.

Thapar demonstrates this failure mode with a real-world enterprise scenario: a user searches for `HP 26X black toner cartridge 2-pack`. The cross-encoder re-ranker reviews the candidates and confidently places a third-party, compatible cartridge at rank 1. The compatible cartridge matches every single extracted specification: it is black, fits the exact same printer series, and delivers the identical 9,000-page yield. However, buried at rank 2 sits the genuine, original OEM HP cartridge. Because the compatible cartridge description shared heavy semantic overlap with the prompt, the ranker favored it, violating the implicit business requirement that an OEM search must prioritize OEM inventory.

```mermaid flowchart LR accTitle: LLM Reasoning Logic for Candidate Disambiguation subgraph Input["Ranker Shortlist"] R1["Rank 1: Compatible HP 26X (Yield: 9k | Non-OEM)"] R2["Rank 2: Genuine HP 26X OEM (Yield: 9k | OEM HP)"] end

subgraph LLM["LLM Attribute Reasoning Audit"] A1["Audit Brand: OEM vs Third-Party"] A2["Audit Yield: 9,000 vs 3,000 pages"] A3["Audit Voltage / Pack Size"] end

subgraph Decision["Final Re-Ordered Match"] F1["Rank 1 Match: Genuine HP 26X OEM"] F2["Rank 2: Compatible Alternative"] end

Input --> LLM LLM --> Decision ```

The Stage 4 reasoning layer solves this by executing a structured, attribute-by-attribute verification audit:

- Evaluates functional **equivalence** rather than mere semantic **resemblance**. - Deconstructs the shortlisted candidates against the canonical query across critical dimensions: OEM brand integrity, page yields, operating voltage, and packaging quantities. - Identifies that Rank 2 is the true intended OEM target and successfully up-ranks it over the compatible alternative. - Constrains LLM token consumption by restricting evaluation exclusively to the top 3–5 items on the re-ranked shortlist. - Serves as the ultimate defense against false positives, enforcing commercial policies that pure vector similarity models inherently miss.

[16:50](https://www.youtube.com/watch?v=IzmgXhrAwcM&t=1010s) Personalization and Multi-Objective Ranking Beyond Traditional Search Metrics

Personalizing product search results introduces a nuanced challenge because there is rarely a single, static definition of what constitutes a "correct" match. A query may yield multiple strictly valid product candidates, but the ideal sorting order depends entirely on the specific individual or business account submitting the request. By tapping into rich historical purchase histories and organizational account profiles, the system can predict which items a customer is most inclined to buy and elevate that specific product to the top spot.

Search engineering teams frequently need to balance user intent with broader business strategies through a multi-objective approach—such as prioritizing private-label own brands or deliberately displaying a diverse spread of manufacturers. Implementing these strategies requires isolating a holdout set of customer traffic to measure impact accurately. Traditional information retrieval metrics like raw click-through rates or NDCG@10 often fail to show the true business value of multi-objective ranking; teams must drill down into margin-weighted outcomes on those holdout cohorts to evaluate success. Blending personalization signals with reasoning models ensures that the system serves commercially viable, context-tailored results without sacrificing strict relevance.

```mermaid flowchart TD accTitle: Multi-Stage Product Matching and Personalization Flow subgraph Retrieval["Stage 1 & 2: Candidate Generation & Ranking"] A["User Query & Account Profile"] --> B["Hybrid Retrieval\n(BM25 + Dual Encoders)"] B --> C["Cross-Encoder Reranking\n(Precise Top-K)"] end

subgraph Reasoning["Stage 3 & 4: Adjudication & Personalization"] C --> D["LLM Equivalence Reasoning\n(False-Positive Reduction)"] D --> E["Personalization & Multi-Objective\n(Margin-Weighted Ordering)"] end

subgraph Feedback["Governance & Continuous Learning"] D -. "Low Confidence Pairs".-> F["Human-in-the-Loop Analysts"] F --> G["Hard Negatives & Active Learning"] G -. "Retrain".-> C end ```

- **Account-level priors:** Historical order frequency and organizational purchasing agreements inform which valid candidate achieves the top rank. - **Multi-objective optimization:** Balances algorithmic relevance against enterprise margin targets and catalog brand diversity. - **Evaluation blind spots:** Standard ranking metrics (such as NDCG@10) do not reflect multi-objective commercial trade-offs, necessitating margin-weighted holdout analysis.

---

[18:05](https://www.youtube.com/watch?v=IzmgXhrAwcM&t=1085s) Balancing Classical ML and GenAI: "Retrieve with Math, Adjudicate with Language"

While generative AI and large language models excel at complex reasoning, their operational overhead remains substantial. Architecting an enterprise-grade search system requires knowing exactly when to rely on traditional machine learning and when to selectively supplement the pipeline with generative components. Classical retrieval mechanisms—such as BM25 for lexical matching and dual encoders for dense semantic vector search—should remain the foundation for global candidate retrieval. Similarly, cross-encoder models provide fast, cost-effective, and highly predictable candidate re-ranking.

Generative models provide the highest return on investment when deployed in two specific interventions: upfront query rewriting to resolve ambiguous or fragmented user inputs, and downstream equivalence reasoning across the refined top-K candidates. Evaluating equivalence at the final stage dramatically suppresses false-positive rates that generic vector search inevitably lets through. The foundational operational guideline is:

> "The rule of thumb should always be retrieve with math, adjudicate with language."

Confining LLM inference strictly to the top-K results keeps API token bills predictable and ensures query response latencies remain within strict production service-level agreements (SLAs).

---

[19:27](https://www.youtube.com/watch?v=IzmgXhrAwcM&t=1167s) Smart Human-in-the-Loop Workflows and the Rise of Agentic Commerce

Maintaining high search precision over time requires continuous refinement via smart human-in-the-loop (HITL) pipelines. When the adjudication layer flags borderline, low-confidence query-product pairs, these ambiguous cases are escalated to human domain analysts. The curated judgments yield high-value hard negatives, which are fed into incremental active learning loops to continually sharpen the re-ranking and retrieval models.

Looking ahead, search architectures are evolving across three major vectors: - **Agentic Retrieval:** Self-correcting query pipelines that dynamically adjust their search strategies based on intermediate retrieval feedback. - **Graph RAG:** Knowledge graph representations that capture deep entity relationships, allowing the system to identify suitable product substitutes when items are discontinued. - **Calibrated LLM-as-a-Judge:** Automated evaluation pipelines calibrated against verified human annotations to benchmark search quality across massive datasets at scale.

These pipelines form the backbone of agentic commerce, where the multi-stage search engine is exposed as a callable tool for autonomous purchasing agents. By structuring complex user constraints—such as explicit budget limits paired with specific part compatibility (for instance, sourcing a specific HP cartridge under a hard price ceiling)—the query understanding layer can extract entities, apply relational constraints, and return exact, policy-compliant matches.

---

[21:10](https://www.youtube.com/watch?v=IzmgXhrAwcM&t=1270s) Core Architectural Takeaways

Reflecting on the design of enterprise product search systems, the session concludes with three essential principles:

1. **Pipelines surpass individual models:** No single foundational model or monolithic embedding space can resolve search by itself. Exact product matching demands an end-to-end multi-stage pipeline. 2. **Hybrid candidate fusion and staged filtering:** Candidate generation requires fusing diverse retrieval strategies, narrowing down candidates via precise cross-encoders, and reserving user personalization for the final stage. 3. **Judicious GenAI adoption with human governance:** Deploy LLMs selectively at points of maximum leverage (query rewriting and top-K semantic reasoning) while continuously reinvesting human review data back into active model training loops.

## Notable quotes > [00:04](https://www.youtube.com/watch?v=IzmgXhrAwcM&t=4s) the only one standing between you and your lunch — Sidharth Thapar

> [04:40](https://www.youtube.com/watch?v=IzmgXhrAwcM&t=280s) Can we find the product at all? — Sidharth Thapar

> [04:40](https://www.youtube.com/watch?v=IzmgXhrAwcM&t=280s) Is the right product at the top? — Sidharth Thapar

> [04:40](https://www.youtube.com/watch?v=IzmgXhrAwcM&t=280s) How often are we catastrophically wrong? — Sidharth Thapar

> [06:22](https://www.youtube.com/watch?v=IzmgXhrAwcM&t=382s) retrieve early for high recall, then rank late for high precision — Sidharth Thapar

> [06:22](https://www.youtube.com/watch?v=IzmgXhrAwcM&t=382s) the most least cool part of the system, but this is the one that actually still solves the real pain in the system that comes later. — Sidharth Thapar

> [07:57](https://www.youtube.com/watch?v=IzmgXhrAwcM&t=477s) We use LLMs only where it's needed to optimize our cost while getting the best results here. — Sidharth Thapar

> [18:05](https://www.youtube.com/watch?v=IzmgXhrAwcM&t=1085s) The rule of thumb should always be retrieve with math, adjudicate with language. — Sidharth Thapar

## Mentioned Sidharth Thapar, Staples Inc., Ai4, HP, HP LaserJet, BM25, ColPali, ColBERT

## Video description Sidharth Thapar - Manager AI / Senior Applied Scientist, Staples Inc.

How can you achieve exact product matches when generic semantic search falls short? I will explain why enterprise needs, like brand-aware substitutions and precise unit handling, require AI that goes beyond simple vector similarity. This talk will illustrate a multi-stage design combining hybrid retrieval, machine-learned ranking, and LLM reasoning for high-precision results. You will learn how to implement human-in-the-loop workflows to refine models and handle complex, ambiguous queries effectively. Join me to discover how these scalable AI pipelines solve nuanced search problems through real-world enterprise implementation.

[Source](https://www.youtube.com/watch?v=IzmgXhrAwcM)

中文简报

Staples 以多阶段 AI 流水线替代纯向量搜索向量嵌入在印量和包装单位等精确规格上存在固有缺陷，使得企业级搜索必须依靠数学模型处理批量候选检索，并将生成式 AI 严格保留用于最终的属性判定。当标准语义模型混淆相似变体（例如混淆标准印量与高印量硒鼓，或误判计量单位）时，企业级商品搜索往往会失效。为解决这一零容忍的挑战，Thapar 概述了一种四阶段架构：通过确定性规则结合选择性 LLM 重写来规范化查询；通过融合多模态门控的 BM25 与稠密向量混合检索召回广泛候选集；通过交叉编码器进行重排；并对排名前列的候选结果利用 LLM 推理验证属性等价性。该流水线践行“用数学做检索、用语言做裁决”的理念，在账户个性化和人机协同主动学习的支持下，平衡了高召回率与计算效率。传统向量相似度搜索优化的是语义相似度而非精确等价性，导致其在企业采购中的零件编号、印量规格及计量单位上存在严重盲区。 (0:04) 企业级商品匹配必须通过单一可配置流水线，满足横跨电商搜索、线下合同报价、定价情报和目录录入等不同组织场景的权衡需求。 (1:15) 搜索工程团队必须将业务 KPI 映射到具体的机器学习指标：以 Recall@K 衡量目录检索完整性，以 NDCG 衡量排序位置，以显式推翻率衡量容错度。 (4:40) 多阶段架构通过遵循核心原则来平衡速度与精度：前期检索追求高召回，后期排序保障高精度。 (6:22) 查询理解采用两步法以控制推理成本：针对常见模式进行确定性规则提取，针对模糊的长尾查询进行选择性 LLM 重写。 (7:57)

Reading summary

Sidharth Thapar of Staples argues that enterprise product matching fails when relying solely on semantic vector similarity, which confuses superficial resemblance with exact equivalence. Addressing this challenge requires a multi-stage architecture that retrieves candidates through hybrid statistical search and adjudicates equivalence using selective language model reasoning. The High Stakes of Enterprise Product Equivalence (0:04) In enterprise procurement, near-miss matches create costly operational failures. Unlike consumer e-commerce, where broad similarity often suffices, business workflows require exact matches across millions of catalog items characterized by cryptic abbreviations, sparse metadata, and non-negotiable business constraints. Standard vector search models optimize for semantic proximity rather than exact identity, creating systematic failure modes. For instance, a query for an `HP 26X` high-yield toner cartridge (yielding roughly 9,000 pages) can easily be misrouted to a standard-yield `HP 26A` (3,000 pages), delivering a third of the expected output. Matching systems must also satisfy divergent cross-functional constraints. E-commerce search requires response times in tens of milliseconds, whereas offline contract quoting can tolerate high latency but demands near-zero matching errors. Pricing intelligence demands explainable feature parity to adjust margins, while catalog onboarding prioritizes broad item coverage. Furthermore, slight unit-of-measure variations—such as a 12-pack versus a single box or a 24-pack—alter pricing by multiples, and policy rules strictly prohibit substituting national brands with private-label alternatives even when physical specifications align. Evaluating Match Quality Beyond General Retrieval Metrics (4:40) Thapar emphasizes that search evaluation frameworks must map directly to business risks rather than generic loss functions. A three-tier evaluation structure benchmarks retrieval completeness, top-position accuracy, and catastrophic failure rates. Retrieval completeness tracks candidate recovery, targeting roughly 98% Recall@100 to ensure the correct target is captured in the initial retrieval pool before narrowing the field. Ranking precision is measured by Normalized Discounted Cumulative Gain, using NDCG@10 for general browsing or NDCG@1 for automated workflows that require the definitive best match in the top slot. To enforce zero-tolerance business requirements, organizations establish an explicit overturn rate budget. This metric determines whether an operation can absorb a small error margin (such as 5%) or requires strict validation checks to prevent false-positive deliveries. Hybrid Candidate Retrieval and Interaction Ladders (6:22) To balance latency and recall across large catalogs, the pipeline pairs deterministic query normalization with hybrid candidate generation. In Stage 1, controlled dictionaries and regular expressions canonicalize shorthand terms (such as mapping `BLK` and `TNR` to `black` and `toner cartridge`), reserving expensive generative query rewriting exclusively for ambiguous long-tail phrasing. Stage 2 combines lexical BM25—which preserves exact alphanumeric part numbers like `CF226X`—with dense semantic embeddings that capture domain synonyms. The two streams are merged via Reciprocal Rank Fusion. For visually ambiguous items where textual confidence drops below a set threshold, confidence gating triggers multimodal models such as ColPali to process spec sheets and product images. Stage 3 sorts the surviving candidate shortlist through an 'interaction ladder.' While bi-encoders evaluate independent embeddings at high speed, late-interaction mechanisms and cross-encoders apply early token-to-token attention to detect minute contextual differences. Because cross-encoders incur quadratic computational overhead, they are restricted strictly to top candidate shortlists rather than applied across the full multi-million-item catalog. Attribute Adjudication, Personalization, and Continuous Learning (15:04) Because cross-encoders score contextual resemblance rather than functional parity, high similarity scores can still endorse flawed matches—such as ranking a third-party compatible cartridge ahead of a genuine OEM cartridge. Stage 4 introduces an LLM reasoning layer to conduct an attribute-by-attribute audit across the top 3–5 items, evaluating critical parameters including OEM status, page yields, packaging quantities, and electrical voltage. Adjudicated items are subsequently reordered using account-level purchase histories and multi-objective business priorities, evaluated via margin-weighted holdout cohorts rather than standard click-through rates. To ensure operational sustainability, low-confidence pairs from the adjudication step are escalated to human domain analysts. This human-in-the-loop review generates hard negatives that feed back into continuous model fine-tuning loops. Thapar notes that this framework supports emerging agentic commerce workflows, enabling autonomous purchasing agents to query catalogs with structured constraints such as compatibility and price ceilings. Thapar concludes with the operational principle to 'retrieve with math, adjudicate with language,' arguing that enterprise search must rely on multi-stage pipelines rather than single monolithic models. Looking forward, architectures are anticipated to incorporate self-correcting agentic retrieval, knowledge graphs for discontinued product substitutions, and calibrated LLM-as-a-judge pipelines to maintain evaluation at scale.

精炼阅读摘要

Staples 的 AI 管理者兼资深应用科学家 Sidharth Thapar 指出，企业级商品匹配的核心难点在于传统语义向量模型仅优化文本相似度，无法满足商业级绝对等价性；系统应采用“以数学召回、以语言裁决”的多阶段流水线，在控制推理成本的同时解决型号、印量和度量单位等零容错匹配难题。业务错配痛点与评估指标对齐 (1:15) 企业商品匹配面临数百万级复杂目录，数据充斥非规范缩写与信息缺失，且企业采购对错配零容忍。例如用户搜索“HP 26X 黑色硒鼓”，26X 代表印量约 9,000 页的高容量型号，若系统误匹配为外观极度相似、仅能打印 3,000 页的标准型号 26A，将导致客户遭受严重的履约缺失。不同业务场景的工程约束截然不同：电商搜索追求数十毫秒响应并允许用户自主过滤；批量报价属于离线计算，要求近乎完美的匹配精度以防合同亏损；定价情报要求特征具有完全的可解释性；商品目录录入则以高覆盖率为核心指标。为避免统计损失函数与实际业务脱节，评估体系必须对应三层检查点：检索完整性优先关注召回率（如目录候选集实现约 98% 的 Recall@100）；排位质量通过 NDCG@10 或针对单品输出的 NDCG@1 进行衡量；针对零容错约束，团队需设定明确的推翻率与错误预算，并引入严格的否决机制以阻断高风险误报。查询理解与多模态混合检索 (7:57) 面对混乱的原始输入，流水线在第一阶段实施两步查询理解：首先通过受控词典与模糊正则提取核心实体（如将品牌 HP、型号 26X 及颜色缩写 BLK 映射为规范词 black）；针对生疏复杂的长尾语法，则选择性触发大语言模型（LLM）进行重写。Thapar 强调，大模型仅在长尾疑难样本中调用，以此在控制推理开销的同时产出高质量的标准化查询。在候选召回阶段，纯文本检索存在固有局限：BM25 擅长精准锁定零件号字符（如 CF226X 中的 26X），却无法识别领域同义词；密集语义向量虽能处理行业别称，但极易抹平单个字符差异而混淆型号。流水线通过互惠排序融合（RRF）整合词法与向量结果，确保标准与高印量型号均进入候选池。当遇到文本极度稀疏（如仅标注“HP CTG BLK”而无法区分墨盒与成像鼓）或复杂规格表时，系统利用置信度门控触发 ColPali 等多模态晚期交互模型，仅在文本置信度不足时解析图像与版面，避免无谓的计算开销。交互阶梯排序与属性等价性裁决 (13:01) 混合检索筛选出数百个候选后，重排序阶段依据“交互阶梯”展开精密排序。模型结构从独立的双编码器、兼顾子词交互与版面的晚期交互（如 ColBERT 或 ColPali），逐步升级至早期全注意力交互的交叉编码器（Cross-Encoder）。交叉编码器具备高度锐利的判别力，但其二次方注意力计算成本极高，只能严格限制在顶部候选集中运行。然而，标量相似度模型仍存在盲区，例如对“12 盒装”与“12 支装”可能打出 0.96 的极高相似分，忽视包装层级带来的倍数级商业差异。为杜绝重排序后的残余误报，第四阶段引入 LLM 执行结构化属性审计。Thapar 举例，当用户检索原装 HP 26X 耗材时，交叉编码器常因文本高度重叠而将规格完全兼容的第三方耗材排在第 1 位，将真正的 OEM 正品压制在第 2 位。LLM 推理层仅审查排名前 3 至 5 项候选，逐一核验 OEM 品牌资质、标称印量、电压及包装规格，从功能等价而非词义相似的角度纠正排序，确保原厂正品置顶。商业多目标优化与自主采购演进 (16:50) 企业商品匹配往往缺乏单一静态的最优解，排序机制需结合客户历史采购偏好与组织协议，在保证核心相关性的同时推行多目标优化（如自有品牌扶持或多品牌展示）。Thapar 指出，通用指标如 NDCG@10 无法体现商业权衡，必须在特定留出样本上评估利润加权成效。系统整体贯彻“以数学召回、以语言裁决”的准则，严格将大模型推理限定在两端，以保障生产级延迟与吞吐预算。为维持长期精度，系统构建了人机协同（HITL）闭环：裁决层标记的低置信度边界样本由人工专家复核，沉淀为高质量硬负例用于主动学习微调。未来该架构正向三大方向演化：具备自校正能力的 Agent 检索、利用知识图谱处理停产替代的 Graph RAG，以及经人工标定校准的 LLM 自动化评测；整套流水线亦可作为标准化工具赋能自主采购智能体，在严格预算与规格约束下执行自动化企业采购。 Thapar 总结强调，单一基础模型或嵌入空间无法独立解决复杂的企业级商品匹配，必须依托层层过滤的多阶段混合流水线；系统的持续可用性则取决于对大模型调用成本的精准控制，以及人机协同数据能否持续回流以驱动检索与排序模型的闭环迭代。

Detailed notes — zh

## Background 该视频由 YouTube 频道 Ai4 于 2026 年 8 月 14 日发布，主讲人为史泰博（Staples Inc.）AI 经理兼高级应用科学家 Sidharth Thapar。视频内容探讨了在通用语义搜索无法满足企业严苛诉求（如品牌替换策略、精确规格与包装单位处理）的背景下，如何构建结合混合检索、机器学习精排与大语言模型推理的多阶段 AI 架构，以实现高精度的商品匹配。 ## TL;DR 在企业级电商和采购场景中，通用向量检索常因追求“相似”而忽略印量、包装单位等关键规格差异，导致高退货率与定价偏差。演讲系统性拆解了一套应对零容错需求的多阶段商品匹配流水线：前期通过规则与 LLM 按需重写规范化查询，中期借助 BM25 与向量稠密检索（辅以多模态兜底及 RRF 融合）确保高召回，后期利用交叉编码器精排并引入大语言模型进行等价性硬逻辑推理与个性化调控。最终，主讲人提出“数学检索，语言裁决”的设计哲学，强调结合人在回路（HITL）闭环与校准后的自动化评估，使系统在迈向智能体电商时代时能够稳步演进。 ## Atmosphere & style 整场演讲呈现出典型硅谷工业界实战派的硬核、严谨与务实风格。主讲人语言精炼、直切痛点，没有空泛吹捧生成式 AI 的神奇魔力，而是冷静剖析不同架构在延迟、算力成本与业务规则硬约束之间的权衡妥协。观众能够深刻感受到来自千万级真实商品库打磨出的工程智慧，随着多阶段漏斗逐步展开，叙事层层递进，最后凝练为三大黄金法则，给人一种极具说服力且条理清晰的听觉体验。 ## Key takeaways - [00:04](https://www.youtube.com/watch?v=IzmgXhrAwcM&t=4s) 通用语义检索在商品匹配上的核心缺陷在于其只寻找“相似商品”，而无法严格保证返回“同一款商品”，微小的规格偏差会导致高退货率与业务损失。 - [02:05](https://www.youtube.com/watch?v=IzmgXhrAwcM&t=125s) 企业内不同场景（在线搜索、报价系统、定价智能、商品编目）在延迟、精度与解释性上存在不同权衡，但需要一套统一可定制的多阶段架构支持。 - [04:40](https://www.youtube.com/watch?v=IzmgXhrAwcM&t=280s) 多阶段商品匹配体系应针对不同阶段设定 KPI，包括底层的 Recall@K 召回率、顶层的 NDCG@1 精排质量以及业务容错底线的翻转率（Overturn Rate）。 - [06:21](https://www.youtube.com/watch?v=IzmgXhrAwcM&t=381s) 端到端的多阶段匹配流水线遵循“先检索保召回、再重排提精度、最后结合推理校验”的设计范式。 - [07:58](https://www.youtube.com/watch?v=IzmgXhrAwcM&t=478s) 查询理解采用两步走策略：首先通过实体识别与词典映射规范化标准词，仅对未覆盖的长尾复杂查询按需调用 LLM 进行重写以控制成本与延迟。 - [09:34](https://www.youtube.com/watch?v=IzmgXhrAwcM&t=574s) 混合检索利用 BM25 解决型号编码硬匹配，利用稠密向量解决领域语义泛化，并通过倒数排名融合（RRF）绕过跨模型分值难以对齐的难题。 - [10:18](https://www.youtube.com/watch?v=IzmgXhrAwcM&t=618s) 针对商品标题极度简略或存在物理部件歧义的边缘场景，系统通过置信度门控按需触发多模态视觉检索，平衡排版细节识别与计算延迟。 - [13:06](https://www.youtube.com/watch?v=IzmgXhrAwcM&t=786s) 在精排阶段，交叉编码器（Cross-Encoder）虽能锐化词符级交互，但对计量单位（UOM）等严格业务硬规则仍缺乏绝对裁决力，需引入逻辑推理层。 - [15:07](https://www.youtube.com/watch?v=IzmgXhrAwcM&t=907s) LLM 推理层用于在 Top-K 候选集上执行“等价性（Equivalence）”深度判定，剔除兼容替代品带来的假阳性，并支持尾端的 B2B 个性化与毛利多目标重排。 - [18:05](https://www.youtube.com/watch?v=IzmgXhrAwcM&t=1085s) 系统架构的核心准则是“用数学检索，用语言裁决”，并通过人在回路（HITL）主动学习将低置信度硬负例反哺回模型流水线。 - [20:09](https://www.youtube.com/watch?v=IzmgXhrAwcM&t=1209s) 智能体电商演进方向包括校准后的 LLM 自动化评估（LLM as a Judge）以及将多阶段检索封装为支持带约束调用的 Agent 工具接口。 - [21:10](https://www.youtube.com/watch?v=IzmgXhrAwcM&t=1270s) 精确商品匹配的三大黄金法则：依赖工程流水线而非单一模型、坚持召回融合与尾端个性化、合理分配 LLM 算力并依托人机协同形成数据飞轮。 ## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=IzmgXhrAwcM&t=4s) 真实业务场景下的商品匹配痛点与挑战在企业级商品搜索与匹配中，用户的输入往往极其简略且充满缩写。例如，用户可能会在搜索框中输入 `HP 26X black TNR CTG 2-pack`（意为惠普 26X 黑色双支装硒鼓/碳粉盒）。普通的搜索引擎可能会召回同类商品，但往往无法精准保证“打印页数印量（Yield）”完全一致——比如用户期望的是能够打印 9,000 页的高容量（High Yield）硒鼓，系统却可能返回了只能打印 3,000 页的标准容量产品。这种差异直接揭示了通用语义搜索在商品匹配上的核心缺陷：**模型被训练用来寻找“相似商品”，而不是严格保证返回“同一款商品”**。在大型企业（如 Staples 史泰博）的实际业务中，商品匹配面对的是千万级庞大商品库，数据本身高度稀疏、格式不一致且充斥着行业缩写（如 `PK` 代表 Pack，`TNR` 代表 Toner）。在这种场景下，数十个极度相似的变体中往往只有“唯一正确”的目标，业务容错率为零（0% Tolerance）。一旦出现轻微偏差（Near Miss）并向客户售出错误规格的产品，将直接引发退货、客户流失以及高昂的履约与售后成本。 ```mermaid flowchart LR accTitle: Product Matching Nuances subgraph Input["用户输入查询"] Q["HP 26X black TNR CTG 2-pack"] end subgraph Failure["通用语义检索误区"] F1["返回 3,000 页标印量 (非 9,000 页)"] F2["返回 24 盒装 (非 12 盒装)"] end subgraph EnterpriseImpact["企业级业务代价"] E1["高退货率与履约损失"] E2["报价/定价策略重大偏差"] end Q --> F1 & F2 F1 & F2 --> E1 & E2 ``` --- ### [02:05](https://www.youtube.com/watch?v=IzmgXhrAwcM&t=125s) 多样化企业用例的权衡与一致性架构需求企业内部存在多种依赖商品匹配的核心场景，不同场景对延迟、精度与解释性有着截然不同的约束与权衡，但企业不应为每个场景构建孤立的烟囱系统，而需要一套兼具灵活性与可定制性的统一架构： - **在线搜索系统（Search Systems）**：延迟是硬性约束，必须在数十毫秒（Tens of milliseconds）内返回结果。该场景对规格的微小偏差容忍度稍高（例如在某些情况下返回 24 包装替代 12 包装是可以接受的妥协）。 - **报价系统（Quoting Systems）**：通常以批处理（Batch）模式运行，对延迟并不敏感，但对匹配的准确率要求极高，必须达到近乎完美的精确匹配。 - **定价智能系统（Pricing Intelligence）**：做出的调价与定价决策必须具备完全的可解释性（Explainable），绝不能仅仅依赖黑盒的向量相似度打分。 - **商品入库编目（Catalog Onboarding）**：核心要求是高覆盖率（Coverage），需要为海量新入库商品尽可能找到匹配项，但往往伴随着精度下降的代价。除了跨场景的指标权衡外，商品匹配还面临着诸多独特的业务细微规则（Nuances）： 1. **严格的排他性规则**：不能用近似品随意糊弄。 2. **品牌硬性策略（Brand Policies）**：例如当用户搜索特定国家知名品牌（National Brand，如 HP 打印机）时，不能随意替换为自有品牌（Private Label）。 3. **单位与包装规格（Units & Pack Sizes）**：在定价场景下，将“12 盒装”误匹为“12 个单盒”会导致价格产生 12 倍的巨大偏差。 4. **大规模工程瓶颈**：高精度的重排模型（如 Cross-Encoder）虽然匹配精准，但计算开销巨大，面对数百万级的高并发查询难以直接落地。 --- ### [04:40](https://www.youtube.com/watch?v=IzmgXhrAwcM&t=280s) 业务 KPI 与多阶段评估指标体系在着手构建技术方案之前，必须建立一套以业务 KPI 为导向、分阶段匹配模型特性的衡量指标体系： | 评估维度 | 核心业务诉求 | 核心评估指标 | 指标定义与业务意义 | |:--- |:--- |:--- |:--- | | **检索召回（Retrieval）** | 目标商品是否能被系统找到？ | **Recall@K**（如 Recall@100） | 衡量底层能否在候选集中覆盖到正确结果，目标通常设定为覆盖 98% 以上的查询。 | | **精排质量（Ranking）** | 正确商品是否排在最前列？ | **NDCG@10 / NDCG@1** | 评估顶层排序质量。如果需要首位绝对精确匹配，则重点考核 NDCG@1。 | | **容错底线（Precision/Tolerance）** | 系统推荐出现错误的比率有多高？ | **错误率 / 翻转率（Overturn Rate）** | 针对 0 容错场景，明确业务是否能接受 5% 以内的误判，评估系统在真实落地时的实际风险。 | --- ### [06:21](https://www.youtube.com/watch?v=IzmgXhrAwcM&t=381s) 整体方案：多阶段商品匹配流水线为了兼顾海量召回、计算效率与极致精度，讲者提出了一套多阶段匹配流水线（Multi-Stage Matching Pipeline）。其核心思想可以概括为：“**先检索保证高召回，再重排保证高精度，最后结合推理进行验证**”。 ```mermaid flowchart TD accTitle: Multi-Stage Matching Pipeline subgraph Stage1["阶段一：查询理解 (Query Understanding)"] A["多模态/文本输入 (文本/图像/文档)"] --> B["实体抽取与规范化字典"] B --> C["LLM 重写长尾非标准词"] C --> D["生成规范化查询 (Canonical Query)"] end subgraph Stage2["阶段二：混合检索 (Hybrid Retrieval)"] D --> E1["字面词法检索 (BM25)"] D --> E2["稠密向量语义检索 (Dense Vector)"] E1 & E2 --> F["候选集粗筛 (Top-N Candidates)"] end subgraph Stage3["阶段三：精确精排与推理 (Ranking & Reasoning)"] F --> G["高精度重排模型 (Re-ranker)"] G --> H["LLM 业务逻辑推理与匹配校验"] H --> I["最终精确匹配结果"] end ``` - **阶段一：查询理解（Query Understanding）**：将非结构化、充满缩写和噪声的原始输入（文本、图像或文档）清洗并转化为系统可解析的标准规范化查询（Canonical Query）。 - **阶段二：候选集检索（Retrieval）**：从数百万级商品库中快速筛选出最相关的 Top-N 候选商品集，重点优化召回率。 - **阶段三：精细排序（Ranking）**：在缩小的候选短名单（Shortlist）上运行计算量更大的精确模型，极大提升排序精度。 - **阶段四：业务逻辑推理（Reasoning）**：超越单纯的文本或向量相似度，利用规则与大语言模型（LLM）推理判断商品属性在业务语义上是否完全契合。 --- ### [07:58](https://www.youtube.com/watch?v=IzmgXhrAwcM&t=478s) 阶段一深拆：两步走查询理解与规范化查询理解虽然在技术直觉上是“最不酷炫的部分（least cool part）”，但却是解决后续所有阶段痛点、决定系统成败的基础设施。系统采用了规则与 LLM 协同的两步走机制： 1. **实体识别与词典映射（Entity Detection & Mapping）**： - 系统首先识别输入中的属性实体，并利用正则表达式（Regex）、模糊匹配（Fuzzy Matching）以及维护受控词表（Controlled Vocabulary），将各种缩写与变体统一映射至标准属性。例如将 `BLK`、`BLACK`、`BLK.` 统一映射为标准 Canonical 词 `black`。 - 目标是将原始碎片化查询（如 `HP 2X 226X BLK TNR CTG two-pack`）规范重构为包含完备属性的标准格式（如 `HP LaserJet toner cartridge high yield 2X black two-pack`）。 2. **LLM 智能重写长尾输入（LLM Rewrite for Tail Queries）**： - 纯规则和硬编码词典无法穷举所有生僻表达。系统仅在遇到未被词典覆盖的长尾（Unseen Tail）查询时才调用 LLM 进行规范化重写。 - 这种按需调用的策略既保证了对未知复杂缩写的泛化解析能力，又最大程度控制了 LLM 的调用成本与延迟。 --- ### [09:34](https://www.youtube.com/watch?v=IzmgXhrAwcM&t=574s) 阶段二深拆：混合检索中字面与语义的互补进入第二阶段的商品检索时，单一的检索技术往往无法同时满足商品型号的硬匹配与行业领域的软理解： - **词法匹配（Lexical Matching / BM25）**：对于包含特定产品型号的查询表现极佳。例如在检索 `26X` 时，BM25 能够精准命中包含 `CF226X` 的具体商品型号。如果直接使用通用的稠密向量嵌入（Dense Vector Embedding）模型，这类关键字母数字编码的精确特征很容易被平滑或丢失。 - **语义匹配（Semantic Matching / Dense Vectors）**：BM25 的短板在于缺乏领域知识（Domain Knowledge）。例如在办公用品领域，“打印墨盒（Print Cartridge）”与“硒鼓/碳粉盒（Toner Cartridge）”在语义上可能表达相同的商品概念，BM25 会因字面不重合而漏召，而语义模型则能很好地捕捉这种同义泛化关系。因此，结合两者的混合检索（Hybrid Retrieval）成为了兼顾型号精准度与语义召回的必经之路。 ### [10:18](https://www.youtube.com/watch?v=IzmgXhrAwcM&t=618s) 混合检索融合与多模态兜底机制在商品匹配场景中，即便使用企业自有的标注数据集对稠密向量（Dense Vector）模型进行微调，模型在语义相同的场景下虽然能给出更精确的结果，但仍难以彻底避免细节混淆的问题——比如当用户明确搜索“26X”大容量耗材时，模型依然可能错误地召回“26A”标准容量耗材。因此，单纯依赖稀疏检索或稠密检索都无法解决根本问题，真正的破局方案在于两者的融合。在工程实践中，通常采用倒数排名融合（Reciprocal Rank Fusion, RRF）算法。RRF 直接在“排名层级（Rank Level）”对两个有序候选列表进行合并，完全绕过了跨模型分值难以对齐和校准（Score Calibration）的难题。这种融合策略在检索阶段（Stage 2）能最大化保证高召回率（High Recall），确保诸如 HP CF226X（高印量）和 CF226A（标准印量）等各版本耗材都能被完整覆盖，至于这些候选项的最终准确排列，则交由后续阶段处理。检索阶段还存在另一类棘手的边界边缘场景：输入文本极其简略（Thin Text），甚至可能存在严重的文本歧义。例如商品标题仅标注为模糊的“HP CTG BLK”，在打印机配件中，硒鼓（Toner Cartridge）与感光鼓单元（Drum Unit）是两种完全不同的物理部件，但它们可能共用完全相同的简写标题，唯有通过产品外观图片才能区分。针对此类问题，系统需要引入多模态检索（Multimodal Retrieval）来协同理解标题与图像，有效剔除假阳性（False Positive）候选项。在架构选型上，双塔编码器（Dual Encoders）速度较快，但在识别包装细微印刷字体（如图片角落标注的 26A 或 26X）时能力有限；而采用晚期交互（Late Interaction）机制的视觉文档模型（如 ColPali）虽然能够深入解析规格表与图文排版，但计算延迟显著增加。为了平衡效果与性能，系统引入了“置信度门控（Confidence Gating）”策略——仅在文本信息稀疏、语义置信度不足时才触发多模态视觉检索，从而避免全量调用带来的高昂算力开销。 ```mermaid flowchart TD accTitle: 混合检索与置信度门控多模态流程 subgraph S1["混合检索召回 (Stage 2)"] A["用户查询与商品数据"] --> B["稀疏检索 (BM25)"] A --> C["稠密向量检索 (Dense Embeddings)"] B & C --> D["倒数排名融合 (RRF)"] end subgraph S2["多模态置信度门控"] D --> E{"文本置信度判定"} E -- "文本充足/置信度高" --> F["标准高召回候选集"] E -- "文本稀疏/存在歧义" --> G["调用多模态/视觉模型 (ColPali/Dual Encoder)"] G --> F end F --> H["进入精排阶段 (Stage 3)"] ``` ### [13:06](https://www.youtube.com/watch?v=IzmgXhrAwcM&t=786s) 交互阶梯与精细化重排阶段的设计与权衡在通过混合检索获得高召回候选集后，系统进入第三阶段：精排（Re-ranking / Stage 3）。精排的目标是在候选集内实现极高精度的排序，通常可以基于单点法（Pointwise）、配对法（Pairwise）或列表法（Listwise）目标进行模型训练。为了捕捉更深层次的词符交互，业内通常借助“交互阶梯（Interaction Ladder）”的不同层级来进行架构权衡： - **无交互的双塔模型（Bi-Encoder）**：虽然计算极快，但无法实现 Query 与 Document 词符级的动态交互。 - **晚期交互机制（Late Interaction）**：以 ColPali 或 ColBERT 为代表，采用词符级最大相似度（MaxSim）计算，在精度与延迟之间取得折中。 - **早期交互的交叉编码器（Cross-Encoder）**：让 Query 与 Document 的每个词符在全注意力层全面交互，能极大锐化语义匹配度，带来极高的精确度。交叉编码器的代价是极高的计算延迟，因此它绝不能直接应用于全库海量候选集，而只能针对检索阶段截断输出的 Top-N 优质候选进行精细打分。然而，即便训练有素的精排模型能够精准区分绝大多数极度近似的变体（Near-Identical Variants），它依然存在盲区——缺乏对严格业务逻辑的“硬性裁决”能力。最典型的例子是计量单位（Unit of Measure, UOM）的硬性约束：当用户搜索“12 box”（12盒装）时，某款“12-pack”（12包装）的商品可能在交叉编码器中计算出高达 0.96 的语义相似度得分。但从电商履约和采购标准来看，规格不符必须被一票否决（Vetoed）。单靠连续分值的相似度排序无法百分之百规避此类业务逻辑假阳性，这也正是后续需要引入逻辑推理层的原因。 --- | 交互架构类型 | 典型代表 / 机制 | 交互时机 | 延迟表现 | 适用场景与局限 | |:--- |:--- |:--- |:--- |:--- | | **双塔模型 (Bi-Encoder)** | 经典 Dense Embedding | 无交互 (独立向量计算) | 极低 (毫秒级) | 全库第一阶段召回；无法捕捉细粒度微小变体差异 | | **晚期交互 (Late Interaction)** | ColPali / ColBERT (Token MaxSim) | 晚期交互 (向量后匹配) | 中等 | 兼顾图文排版细节与一定检索吞吐 | | **交叉编码器 (Cross-Encoder)** | Listwise Cross-Encoder | 早期全注意力交互 | 高 | 仅适用于 Top-N 候选精排；但对业务硬规则 (如 UOM) 缺乏绝对判断力 | --- ### [15:07](https://www.youtube.com/watch?v=IzmgXhrAwcM&t=907s) 大语言模型推理、等价性判定与业务多目标个性化第四阶段（Stage 4）的核心是引入大语言模型（LLM）的生成式推理能力，弥补传统精排模型“知其相似而不知其等价”的缺陷。传统重排模型依据历史数据学习到的属性特征和分值进行排序，往往容易将表面属性齐全但本质不对等的替代品推至首位。例如用户搜索“HP 26X 黑色双支装硒鼓”，精排模型很可能将一款标有“Compatible with HP 26X”（兼容款、印量同为 9000 页）的第三方耗材排在第一名，而将真正的原装 HP 耗材排在第二名。LLM 推理阶段在此处发挥的关键作用是：不仅比对字面“相似性（Resemblance）”，更逐一拆解关键属性（如印量、工作电压、原厂与兼容属性）进行深度“等价性（Equivalence）”判定，从而将原厂正品精准重排至第一位，大幅削减高分假阳性率。由于 LLM 推理伴随 Token 成本和延迟，该步骤仅严格作用于精排后输出的极小集合（Top-K）。在推理验证的基础上，系统还可在本阶段叠加“个性化（Personalization）”与“多目标重排（Multi-Objective Ranking）”逻辑： ```mermaid flowchart LR accTitle: LLM推理与多目标精细化决策流 subgraph In["输入候选"] A["Top-K 精排候选集"] end subgraph LLM["LLM 等价性推理"] B["属性逐项核验 (印量/原厂兼容/UOM)"] C["假阳性剔除与正品提权"] end subgraph MultiObj["业务目标与个性化调控"] D["历史采购偏好 (B2B客户账号画像)"] E["自有品牌/毛利加权 (Margin-Weighted Holdout)"] end subgraph Out["最终呈现"] F["最终最优排序结果"] end A --> B --> C C --> D & E --> F ``` - **个性化排序**：搜索结果中往往存在多个完全合规的有效匹配项，但不同 B2B 客户基于历史采购订单或企业账户偏好，其最佳第一位推荐完全不同，需要结合用户画像动态调序。 - **多目标权衡**：企业在实际运营中往往需要推广自有品牌（Own Brands）或丰富品牌多样性。在评估此类多目标调整时，传统的点击率（Clicks）或 NDCG@10 往往无法全面反映收益，必须在预留对照组（Holdout Sets）上深入分析毛利加权指标（Margin-Weighted Results）。 ### [18:05](https://www.youtube.com/watch?v=IzmgXhrAwcM&t=1085s) “数学检索，语言裁决”原则与人在回路系统演进在当前生成式 AI 时代，必须清晰界定传统机器学习（Classical ML）与大语言模型（GenAI）的分工边界。传统检索模型（如 BM25、双塔编码器）与精排交叉编码器具备廉价、高效、输出可预测的绝对优势，是提升全库召回率和粗排效率的主力；而 LLM 则应聚焦于其核心强项——利用 Query 改写（Query Rewriting）处理模糊歧义的自然语言，以及在最终 Top-K 阶段执行深度的“等价性推理（Equivalence Reasoning）”以降低假阳性。针对系统架构设计的核心准则是： > “永远坚持用数学来检索，用语言来裁决（Retrieve with math, adjudicate with language）。” 为了使整个模型流水线持续进化，系统必须构建一套智能的“人在回路（Smart Human-in-the-Loop）”闭环体系。当精排与 LLM 判定出现低置信度（Low Confidence）结果或模棱两可的边缘案例时，系统会将样本路由给人工分析师（Analysts）进行研判标注。这些高质量的硬负例（Hard Negatives）将通过增量主动学习（Incremental Active Learning）反哺到早期的检索与精排模型中，实现端到端的持续迭代。在未来的技术演进路线上，企业级商品搜索与匹配体系还可以向三个方向进一步扩展： - **智能体检索（Agentic Retrieval）**：赋予系统自主规划检索策略与自纠错（Self-Correction）的能力； - **图检索增强生成（Graph RAG）**：利用实体关系图谱精准刻画复杂的商品兼容矩阵与层级属性； - **大模型即裁判（LLM as a Judge）**：构建自动化的大规模端到端离线评估流水线，持续保障检索与排序系统的工业级精度。 ### [20:09](https://www.youtube.com/watch?v=IzmgXhrAwcM&t=1209s) 规模化评估与智能体电商架构演进在讨论商品停产与替代品关系时，系统能够依据预先建立的关联知识库，在商品下架或停产时精准推荐合理的替代品。对于整体系统的性能评估，如果想利用 "LLM as a judge"（大模型作为裁判）实现规模化评测，核心前提是必须先针对自有业务数据进行充分的校准（calibration）。只有完成对齐与校准，大模型裁判才能自主且准确地执行评估，从而实现大规模数据的自动化验证与质量监控。与此同时，当前的电商搜索正快速迈向“智能体电商”（Agentic Commerce）新范式。理想的架构是构建一套端到端的端式多阶段系统：一方面实现智能体自主检索（Agentic Retrieval），另一方面必须将搜索系统封装为外部 AI Agent 可以直接调用的 Tool（工具）。在此场景下，查询理解（Query Understanding）需要进一步升级以处理复杂的结构化约束和预算条件——例如当用户输入“我想买惠普（HP）墨盒，预算是 XYZ”时，系统能够精准推理并抽取特定实体，自动将价格预算等硬性约束转化为检索过滤条件。 ```mermaid flowchart TD accTitle: 智能体电商与多阶段检索架构 subgraph S1["查询与推理层"] A["复杂用户查询 (含品牌/预算等约束)"] --> B["智能查询理解 & 实体抽取"] B --> C["工具调用接口 (Tool Interface for Agents)"] end subgraph S2["执行与评估层"] C --> D["多阶段混合检索"] D --> E["业务约束过滤 & 重排"] E --> F["校准后的大模型自动化评估 (LLM as Judge)"] end ``` - **替代品关系处理**：在商品停产时，系统能够基于实体关系图谱快速无缝切换至匹配的替代品。 - **LLM 裁判校准**：大语言模型在用于系统评估前，必须结合自身领域数据进行标定，确保自动打分与人类专家判断一致。 - **Agent 工具化对接**：搜索检索管线需要对外提供标准化工具接口，支持 AI 智能体带着约束条件直接发起调用与逻辑推理。 --- ### [21:10](https://www.youtube.com/watch?v=IzmgXhrAwcM&t=1270s) 核心总结：精确商品匹配的三大黄金法则在演讲的结尾，Sidharth Thapar 提炼了三条最为关键的实践总结，强调即使观众忘掉其他所有细节，也必须牢记这三点原则：首先，在任何搜索系统中，“没有单一模型能够通赢全场”（"No single model really wins in any search system"）。实现高精度的商品精确匹配，靠的是一套严密的工程 Pipeline（流水线），而不是仅仅依赖单一的向量嵌入（Embedding）。其次，必须坚持在召回阶段采用多路结果融合（Hybrid Fusion）以生成全面的候选集，随后接入高精度的精排模型（Re-ranker）筛选出 Top-K 结果，而个性化（Personalization）层应当放在流水线的最后一步生效。最后，构建长效进化系统时，必须始终引入人机协同（Human-in-the-Loop, HITL），并“仅在真正需要的大模型推理阶段使用 LLM”。将团队投入的高质量时间通过 HITL 标注或针对性推理进行复利累积，才能让电商搜索与匹配系统随着时间推移持续进化。 - **法则一**：精确商品匹配依赖端到端多阶段流水线，单一向量检索无法解决复杂的商业与物理属性匹配。 - **法则二**：召回多路融合 $\rightarrow$ 精度精排截断 Top-K $\rightarrow$ 尾端个性化（Personalization comes last）。 - **法则三**：合理权衡算力与效果，将大模型用在关键推理节点，同时依托人机协同机制形成数据飞轮。 ## Notable quotes > [00:04](https://www.youtube.com/watch?v=IzmgXhrAwcM&t=4s) 模型被训练用来寻找“相似商品”，而不是严格保证返回“同一款商品”。 — Sidharth Thapar > [02:05](https://www.youtube.com/watch?v=IzmgXhrAwcM&t=125s) 在某些情况下返回 24 包装替代 12 包装是可以接受的妥协 — Sidharth Thapar > [02:05](https://www.youtube.com/watch?v=IzmgXhrAwcM&t=125s) 在定价场景下，将“12 盒装”误匹为“12 个单盒”会导致价格产生 12 倍的巨大偏差。 — Sidharth Thapar > [06:21](https://www.youtube.com/watch?v=IzmgXhrAwcM&t=381s) 先检索保证高召回，再重排保证高精度，最后结合推理进行验证 — Sidharth Thapar > [07:58](https://www.youtube.com/watch?v=IzmgXhrAwcM&t=478s) 最不酷炫的部分（least cool part） — Sidharth Thapar > [18:05](https://www.youtube.com/watch?v=IzmgXhrAwcM&t=1085s) 永远坚持用数学来检索，用语言来裁决（Retrieve with math, adjudicate with language）。 — Sidharth Thapar > [21:10](https://www.youtube.com/watch?v=IzmgXhrAwcM&t=1270s) No single model really wins in any search system — Sidharth Thapar > [21:10](https://www.youtube.com/watch?v=IzmgXhrAwcM&t=1270s) 实现高精度的商品精确匹配，靠的是一套严密的工程 Pipeline（流水线），而不是仅仅依赖单一的向量嵌入（Embedding）。 — Sidharth Thapar > [21:10](https://www.youtube.com/watch?v=IzmgXhrAwcM&t=1270s) 仅在真正需要的大模型推理阶段使用 LLM — Sidharth Thapar ## Mentioned Sidharth Thapar, Staples Inc., Ai4, HP, BM25, ColPali, ColBERT, Reciprocal Rank Fusion (RRF), Cross-Encoder, Bi-Encoder, Graph RAG ## Video description Sidharth Thapar - Manager AI / Senior Applied Scientist, Staples Inc. How can you achieve exact product matches when generic semantic search falls short? I will explain why enterprise needs, like brand-aware substitutions and precise unit handling, require AI that goes beyond simple vector similarity. This talk will illustrate a multi-stage design combining hybrid retrieval, machine-learned ranking, and LLM reasoning for high-precision results. You will learn how to implement human-in-the-loop workflows to refine models and handle complex, ambiguous queries effectively. Join me to discover how these scalable AI pipelines solve nuanced search problems through real-world enterprise implementation. [Source](https://www.youtube.com/watch?v=IzmgXhrAwcM)

Source quotes

[0:04] Sidharth Thapar: 模型被训练用来寻找“相似商品”，而不是严格保证返回“同一款商品”。 [2:05] Sidharth Thapar: 在某些情况下返回 24 包装替代 12 包装是可以接受的妥协 [2:05] Sidharth Thapar: 在定价场景下，将“12 盒装”误匹为“12 个单盒”会导致价格产生 12 倍的巨大偏差。 [6:21] Sidharth Thapar: 先检索保证高召回，再重排保证高精度，最后结合推理进行验证 [7:58] Sidharth Thapar: 最不酷炫的部分（least cool part） [18:05] Sidharth Thapar: 永远坚持用数学来检索，用语言来裁决（Retrieve with math, adjudicate with language）。 [21:10] Sidharth Thapar: No single model really wins in any search system [21:10] Sidharth Thapar: 实现高精度的商品精确匹配，靠的是一套严密的工程 Pipeline（流水线），而不是仅仅依赖单一的向量嵌入（Embedding）。 [21:10] Sidharth Thapar: 仅在真正需要的大模型推理阶段使用 LLM

Source quotes

[0:04] Sidharth Thapar: the only one standing between you and your lunch [4:40] Sidharth Thapar: Can we find the product at all? [4:40] Sidharth Thapar: Is the right product at the top? [4:40] Sidharth Thapar: How often are we catastrophically wrong? [6:22] Sidharth Thapar: retrieve early for high recall, then rank late for high precision [6:22] Sidharth Thapar: the most least cool part of the system, but this is the one that actually still solves the real pain in the system that comes later. [7:57] Sidharth Thapar: We use LLMs only where it's needed to optimize our cost while getting the best results here. [18:05] Sidharth Thapar: The rule of thumb should always be retrieve with math, adjudicate with language.

Original transcript

[0:04] Someone searches HP 26X [0:07] black TNR CTG, that's a toner cartridge [0:12] and a pack of two. It's at the bottom [0:14] left of your screen. I'm sorry you guys [0:16] can't see it properly, but [0:18] that's a product search a particular [0:20] customer makes and a typical search [0:22] engine might return the same product, [0:24] but it might not always return the same [0:26] yield. So, let's say the toner cartridge [0:29] produces [0:30] uh is able to print 9,000 pages, but it [0:34] might return a product that returns only [0:35] 3,000 pages. [0:37] That gap is exactly the talk is all [0:39] about. [0:41] And it's not about the model. We can't [0:43] blame the model cuz model is trained to [0:44] return similar products, [0:46] but it's not always trained to return [0:48] the same product. [0:51] Hi everybody, I'm Sidharth Thapar. [0:53] I work in AI. I build recommendation [0:56] systems, search systems, [0:58] and I'm the only one standing between [1:00] you and your lunch. So, please hold on [1:03] uh for the next 20 minutes. [1:05] Uh look at that bottom query. Uh that's [1:08] going to be the entire talk and how we [1:10] can improve our search system and build [1:12] something that's robust. [1:15] Moving on, I want to first set the stage [1:18] with talking about why product matching [1:20] is hard. [1:21] Typical search systems search from large [1:24] catalogs where [snorts] the in the input [1:26] data is sparse, it's inconsistent, it's [1:29] abbreviated heavily the way we saw the [1:32] previous example where we see a PK and a [1:34] TNR for a toner pack. [1:37] And typically, you always have one [1:40] correct product that you want to return [1:42] or one correct result that you need to [1:44] return out of the dozens of variations [1:46] of that same result. [1:49] And then, there is a 0% tolerance. If [1:52] you have a near miss, [1:54] it ends up returning a wrong product [1:56] that is sold and it might it might have [1:58] costs like you the customer might return [2:00] the product. So, [2:02] that is why this problem is hard. [2:05] And the idea is not about building [2:07] customized solutions for every problem. [2:10] In a typical enterprise, you would see [2:12] search, quoting systems, pricing [2:15] intelligence, catalog onboarding. These [2:17] are different some examples of different [2:19] types of use cases where you need to [2:21] find matches to your products or any [2:24] search system that you're building. [2:26] But, check the nuance. Search systems [2:28] typically require tens of milliseconds [2:30] of return time, so the latency is a [2:31] constraint, but they generally might [2:34] return a 24-pack for a 12-pack. [2:36] So, they are they are fine with that [2:38] trade-off, but latency is the primary [2:40] constraint. For a typical quoting [2:42] system, you could run or generate a [2:44] quote in a batch in a batch, so the [2:46] latency is not a concern, but you need [2:48] to be precise or require near-perfect [2:51] matches. [2:53] On the other hand, if you look at [2:54] pricing systems, when you're making a [2:56] pricing decision, the decisions must be [2:58] explainable. They cannot be just based [3:00] on a similarity score. [3:02] And let's let's think about catalog [3:04] onboarding. When you're trying to [3:07] you definitely need high coverage. You [3:09] want to be finding pro- results or [3:12] matches for everything you want to [3:13] search, but that comes at a cost of [3:16] low precision. [3:18] So, these are the different types of [3:21] challenges that enterprises face, and [3:23] the idea is not about building different [3:25] systems. It's about building one single [3:27] system and then allowing it with a [3:29] flexible design that makes it [3:30] customizable for the different use cases [3:32] you might have across your enterprise. [3:35] Let's also talk about why this problem [3:37] is nuanced. [3:38] You end up trying to solve for [3:41] exactness, where you cannot return a [3:44] similar cousin. Say a 24-pack might not [3:46] make sense for your use case for a [3:48] 12-pack that you're searching. [3:50] Similarly, you might have hard-coded [3:52] policies like you want to return a [3:54] private label product for a specific [3:56] national brand. Let's say you cannot [3:58] return a private label printer for the [4:01] HP printer that you're searching. [4:03] You might have very specific [4:04] requirements on units and pack sizes. [4:07] Going back to the pricing example that I [4:08] talked about, you cannot return a 12 box [4:12] pack for a 12 box when you're searching [4:14] because that is 12 times in prices and [4:16] that may So, these rules might require [4:19] you to be hardcoding them and there's a [4:21] limit to how much you can hardcode. [4:23] And then similarly, there's a scale [4:25] problem at hand. There are precise [4:27] re-rankers, precise ranking systems like [4:30] cross-encoder models, but they are much [4:33] slower than what your needs are. They [4:35] need They're too slow for millions of [4:37] queries, typically speaking. [4:40] So, before we go into how to solve this, [4:42] let me set the stage of how do you [4:43] measure the performance of the system [4:45] that you're building here. [4:47] And the idea here that I want to talk [4:49] about here is you need to focus on what [4:51] business KPI you're targeting and then [4:53] choose the model metrics accordingly. [4:55] For example, let's look at the first [4:56] one. [4:57] The first question business is typically [4:59] going to ask is can we find can we find [5:02] the product at all in the results? That [5:05] basically means you need to focus on the [5:07] recall and something like recall at 100 [5:10] could be a good choice where you would [5:12] want to measure are we typically able to [5:14] answer 98% of our queries or not. [5:16] But when you're answering the query, it [5:18] might not always appear at the top and [5:20] that's where your second system comes in [5:22] where you want to make sure the ranking [5:24] quality is also good and it's just not [5:26] present in the results. [5:28] And that is where the decision making [5:30] might be different. So, you might [5:31] recommend a very similar product at top [5:33] and there are downsides of having uh [5:37] a wrong a wrong recommendation for [5:38] something that you would want uh it to [5:41] be at rank one. So, an NDCG is a good [5:44] choice there where you would want to see [5:45] if it is returned in the top 10 results [5:47] or at NDCG at one if you want the [5:50] precise result at the top. [5:52] But then, you might return the results [5:55] at the top rank, but how often are we [5:57] wrong? [5:58] And that is that takes us to the [6:00] zero-tolerance constraint that we talked [6:02] about in the beginning. [6:04] You have to understand how what is the [6:06] tolerance of your system. Are you [6:07] comfortable with a 5% overturn? [6:11] And that's how you need to make sure as [6:13] you deliver these systems, you're able [6:15] to measure them effectively and [6:17] understand if they actually solve a real [6:19] business problem. [6:21] Let's move on to the actual solution [6:22] now. [6:23] What I'm proposing here is a multi-stage [6:25] matching pipeline. [6:27] The idea is to first understand the [6:29] input data. The input data could be a [6:32] text field, it could be an image, it [6:35] could be a document. [6:37] But the idea is to first understand that [6:39] and convert that into a one canonical [6:41] query, [6:42] which is digestible for your system. [6:45] This is the most least cool part of the [6:48] system, but this is the one that [6:49] actually still solves the real pain in [6:51] the system that comes later in the [6:53] future stages. The second stage is going [6:55] to be about how do you retrieve the most [6:57] relevant candidates. [6:59] A typical system wants precision, but [7:02] you cannot build the most precise [7:04] solution as we talked about the [7:05] cross-encoders, for example, to just [7:08] solve [7:09] to find the most relevant product out of [7:10] millions of documents that you're [7:12] searching against. So, retrieval helps [7:14] you find the most relevant candidates, [7:17] find your top end, and then move towards [7:20] ranking where you're able to [7:22] give more precise results. So, you have [7:24] high recall, but you want to now work on [7:26] a smaller list or short list where you [7:28] want to go more precise. [7:30] And then, once you have a precise [7:32] ranking results, [7:34] a typical use case is not about just the [7:36] similarity of the similarity of the two [7:39] titles or the input data versus the [7:40] query. [7:41] It's about reasoning it if it really [7:44] matches and solves your problem. [7:46] So, we'll talk more about this stage by [7:48] stage. [7:49] But, the takeaway here is retrieve first [7:52] for high recall and re-rank it for high [7:55] precision. [7:57] Let's talk about the stage one now, [7:58] which is query understanding. Think of [8:00] it as two-step process. What we [8:03] typically do is we first detect all the [8:05] different attributes. There's an example [8:07] at the bottom that you see, HP LaserJet [8:10] toner cartridge high yield 2X black [8:12] two-pack. That's the version that we [8:14] want to achieve, but the typical query [8:16] is similar to HP 2X 226X BLK TNR CTG [8:21] two-pack. [8:22] So, what we end up doing is we identify [8:25] the entities within your input query, [8:28] and we try to map all of the different [8:30] variations of that. Let's say BLK, [8:32] BLACK, BLK. We want to map all of those [8:36] to a canonical object and call that [8:38] black. [8:39] We typically do that by maintaining a [8:41] dictionary, or we use some fuzzy fuzzy [8:45] matching or regex methodologies to [8:47] achieve it. But, there's a limit to how [8:49] much you understand your data. [8:51] And that is where the second step LLM [8:53] rewrite helps you. So, we have to be [8:55] smart, and I think that's the theme of [8:56] the entire top entire topic here. LL We [8:59] use LLMs only where it's needed to [9:01] optimize our cost while getting the best [9:04] results here. [9:05] And LLM rewrite helps you work on that [9:07] unseen tail. So, all of the scenarios [9:10] that you couldn't capture in your [9:11] hardcoded [9:12] dictionaries or through the fuzzy [9:14] matching system is where you'll be able [9:17] to help rewrite and map it to a [9:19] canonical object. [9:22] Now, the cost here is still a controlled [9:24] vocabulary has to be maintained. That's [9:26] the cost you're getting, but you're able [9:28] to solve the ambiguous abbreviated input [9:31] to something that is standardized. [9:34] We'll move on to the stage two now, [9:36] where we talk about how do we retrieve [9:38] our input data. [9:40] Assuming our input data is text, [9:42] we first work towards [9:44] a very popular technique that's [9:46] standardized over the last decade using [9:48] lexical matching. A typical algorithm [9:50] could be a BM25, which does really well [9:53] when you're trying to find a 26X for a [9:56] CF226X. [9:58] That is the model number of the product [10:00] that you're searching. So, that [10:01] survives, but that wouldn't have [10:03] survived if it was a typical embedding [10:04] model, a dense vector model. [10:07] But, it does not do well when a print [10:10] cartridge is the same as toner [10:12] cartridge. That's your domain knowledge, [10:14] and that is what a similarity model, [10:16] semantic similarity model, or a dense [10:18] vector can get you. So, if you fine-tune [10:20] your dense vector model on uh your uh [10:23] labeled data set from whichever [10:25] enterprise you're working on, it gets [10:27] you precise results on where the meaning [10:30] is the same, but it will and if you [10:32] notice, it still fails in the scenarios [10:34] of returning a 26A for a 26A for a 26X [10:38] that you searched. [10:40] So, the solution here is neither. It's [10:42] actually a fusion of these two, and a [10:44] typical algorithm [10:46] uh to achieve that is reciprocal rank [10:47] fusion. What that does is that it merges [10:50] the two rank lists, and it merges them [10:53] at the rank level and not at the score [10:54] level. So, you don't try to compute a [10:57] calibrated score. You actually just [10:59] merge those two lists. [11:01] And once you have those two lists, it [11:03] gives you what you were looking for [11:04] overall in the retrieval stage, which is [11:06] high recall. Both [11:08] uh CF2226X [11:11] and A, both of those, which are [11:13] high-yield and your low-yield variants [11:15] of that toner cartridge, are available [11:17] in our results. But, they might not [11:19] necessarily be ranked at the correct [11:20] location, and that's where stage three [11:22] comes in. [11:23] But, before I move on to the stage [11:24] three, I just want to talk about an edge [11:26] case scenario where the input data might [11:28] not always be text, or the text might be [11:30] too thin. For example, let's look at the [11:33] bottom at the bottom of the slide where [11:34] you see HP CTG BLK. We might have a [11:37] toner cartridge and a drum unit. Both [11:40] are different parts of a printer, but [11:43] they might share the same title. They [11:45] might ship with the same title, but they [11:47] have different product images. So, a [11:49] multimodal retrieval can help you [11:51] understand based on the title and the [11:53] image of the product and find the find [11:56] these false positive scenarios and only [11:58] return relevant products as your [12:00] candidates. [12:01] Dual encoders, for example, are the [12:03] typical [12:04] solutions here because generic encoders [12:08] read Let's say if you have two images [12:10] and the fine print on that specifies the [12:12] model number like 26A or 20 [12:14] 26X. [12:16] That is not what a dual encoder would [12:18] would catch normally. So, you need to [12:19] find You need using the same strategies [12:21] we talked about. [12:22] And then there could be another solution [12:24] where you have visual documents, where [12:26] you have a document where you have [12:27] specification mentioned, you have [12:29] tables, you have images. And something [12:31] like a Coal Pali can be used, which uses [12:35] late interaction mechanism. [12:37] But then it is slower than a dual [12:38] encoder, so it doesn't really solve the [12:40] overall problem here. So, the idea is to [12:43] use confidence gating here. Confidence [12:45] gating is as simple as invoking vision [12:48] only when text is not reliable. So, you [12:50] you only use image-based retrieval or [12:52] multimodal retrieval when your input [12:55] text is thin. [12:56] And that is a great example of what I'm [12:58] trying to say here. So, let's move on to [12:59] stage three now, which is once you have [13:02] high recall through your retrieval [13:03] models, how do you achieve precise [13:05] ranking? [13:06] You typically go for a pointwise, a [13:09] pairwise, or a listwise objective, and [13:12] you want to train your listwise reranker [13:14] with all of the previous scenarios that [13:16] you've seen. [13:17] What happens is when you use something [13:20] like a cross encoder, you're able to use [13:22] early interaction, which sharpens your [13:24] results. So, you do get high accuracy, [13:26] but it comes at a cost of [13:29] uh low latency. So, you need to only [13:31] work on the top end retrieved [13:33] candidates. You cannot work on billions [13:34] of queries. That's the catch here. [13:37] But, then [13:38] there are different ways you can still [13:39] design your re-ranker. You can If you [13:41] look at the interaction ladder in the [13:42] middle, you have a bi-encoder which does [13:45] not have interaction. You have a late [13:47] interaction mechanism when you use [13:49] something like uh from the previous [13:51] slide, if you remember, the ColPali [13:53] models use a portal token maximum [13:56] similarity methodology, which gives you [13:57] more precise results, but it is a slower [13:59] methodology. And then the And then the [14:01] more advanced method of a cross-encoder, [14:03] which does early interaction, but it is [14:05] even slower for and that's the [14:07] compromise for sharper, precise ranking [14:10] results. [14:11] But, but I want to highlight another [14:13] thing that this gets you to a precise [14:16] real high recall and precise results, [14:18] but it still misses some edge cases. If [14:21] you look at the bottom example there, [14:22] unit of measure must exactly match, [14:25] whatever the similarity says. A 12-pack [14:28] scoring a high score like 0.96 against a [14:31] query for a 12 box is vetoed. We need a [14:34] system that takes care of this, and a [14:36] high score does not guarantee, even on [14:39] the ranking, that you would catch these [14:41] scenarios. [14:43] And and let's let's move on to the next [14:45] slide uh of stage four, but before that [14:47] I want to just call out that this fixes [14:49] your near identical variants problem. [14:52] You're able to identify all your typical [14:54] near misses, but you're still and but it [14:57] comes at a cost of deep training and [14:59] latency, and that's why a combination of [15:02] stage two and three makes sense. [15:04] Once you're able to get precise ranking [15:06] as well, [15:07] I want to take an example here to to [15:09] demonstrate why you need reasoning. And [15:11] this is where GenAI is brought in again, [15:15] because reasoning is where it is super [15:17] effective. So, the ranker [15:19] puts the the list first confidently, [15:21] because it's doing it based on its [15:23] scoring on all of the attributes and the [15:25] nuances it's learned from your data. But [15:27] it does not reason on examples like [15:30] these. For Let's look at that. The same [15:33] query HP26X black toner cartridge [15:35] two-pack. [15:37] The your typical re-ranker might return [15:39] a compatible with that particular title [15:42] is your first product as rank one. It is [15:44] the correct yield. It's still 9,000 [15:46] pages as we see. So every attribute [15:48] matches, but you still have an [15:51] irrelevant product returned while you [15:52] had the second product in your [15:54] vocabulary, in your in your knowledge [15:56] base. So you should have returned the [15:58] second product as the first. [16:00] So the idea here is we use an LLM [16:01] reasoning approach where you look for [16:03] not just resemblance, you also look for [16:05] equivalence. And you want to go [16:08] attribute by attribute. You go deep into [16:10] one by one on understanding what's the [16:12] yield, what's the voltage of the HP [16:15] cartridge, and that gets you to [16:17] up-ranking your second rank as the first [16:19] match with this reasoning. [16:22] This obviously comes at tokens cost. [16:24] That is why we only work on the small [16:26] re-ranked precise list. And this is the [16:29] stage which is going to help you [16:31] reduce your false positive percentage [16:33] that we talked about. Because these [16:35] systems are great at ranking till stage [16:38] three, but they're not typically good at [16:40] identifying false positives. That's [16:42] where this is more valuable than the [16:44] other stages. [16:45] And within the same stage you could also [16:47] think of Sometimes you might need [16:48] personalized results. Let's think of an [16:50] example of when [16:52] uh [16:54] Personalizing the order of valid matches [16:57] is never the definition of a correct [16:59] one. So you might want to uh personalize [17:03] your results in a like you might have [17:05] more than one valid results, but the [17:07] results in the specific order might be [17:09] very specific to the user that needs it. [17:13] And you you might have a lot of data. If [17:15] you go to the second one here, you might [17:16] have a lot of data based on the previous [17:18] orders a customer has placed or what [17:20] what account they come from. That would [17:22] show you what products are they likely [17:24] to buy and that might highlight which [17:26] valid product needs to be ranked one. [17:29] And sometimes you might want to go [17:30] towards a multi-objective approach of [17:32] let's say you want to sell your own [17:34] brands or you want to show a diverse [17:36] brand result. [17:37] Those are the cases where you might want [17:39] to hold out a sub certain set of [17:41] customers where you want to perform [17:42] this. And typically your clicks or your [17:46] NDCG at 10, your ranking quality and [17:49] your retrieval quality might not show [17:51] the benefit of multi-objective. You need [17:53] to go deeper and look at margin weighted [17:55] results for that holdout. So that is why [17:58] combining personalization and LLM [18:00] reasoning can get you to the exact [18:02] results that you're looking for. [18:05] Now now we've talked about the [18:06] multi-stage approach, but I just want to [18:08] highlight this topic as we are we're in [18:10] the era of GenAI and these typical LM [18:13] system LLM systems are great at doing [18:15] things, but they come at a cost. So we [18:18] want to know when we want to use ML [18:20] systems and when we want to supplement [18:22] them with GenAI. As we discussed a few [18:25] examples here, let's keep it classical [18:27] when we're trying to when we're trying [18:29] to increase our global recall with [18:30] retrieval systems like BM25, dual [18:32] encoders or when we're when we're trying [18:34] to get precise [18:36] reranking results through cross [18:37] encoders, they're cheap, they're fast, [18:39] they're predictable. But when you're [18:41] trying to [18:42] uh cover up for the ambiguous titles as [18:45] we discussed, query rewriting is where [18:47] it can help you understand the query [18:49] even better. Uh so that your ML systems [18:51] perform better. [18:53] We also talked about equivalence [18:54] reasoning of the top K. [18:56] The reasoning after you have your [18:58] precise results, all the valid results, [19:00] it helps you reduce your false [19:01] percentage. So those are the two key [19:03] areas where you can use it. [19:05] And [19:06] one thing I want you to catch is the [19:07] rule of thumb should always be retrieve [19:09] with math, adjudicate with language. [19:13] Your stage four top K sees only the top [19:15] K results, but the both the latency and [19:19] the bill needs to be same and the token [19:21] cost is what you're paying for every [19:23] system that uses LLM. [19:27] We'll move on to a smart human in the [19:28] loop approach. [19:30] Uh where low confidence pair analysts [19:33] look at the look at the ambiguous [19:34] situations and then you use those hard [19:36] negatives to use incremental active [19:40] learning to actually improve your system [19:42] incrementally. [19:45] I think there are some more interesting [19:46] ideas that I wanted to share, but feel [19:48] free to go through my slides. I think [19:50] I'm [19:51] almost out of time now, but you can [19:53] evolve these systems through agentic [19:56] retrieval, [19:57] through graph rags, and then you can [19:59] build an LLM as a judge to eval at [20:02] scale. [20:04] LLM agentic retrieval self-corrects [20:06] themselves. Graph rags help you identify [20:09] substitute relations when when products [20:11] they discontinue [20:13] because they establish relationships. [20:15] And then evaling at scale using LLM as [20:17] judge needs to be first calibrated to [20:19] your data so that it can actually do [20:21] evaluation on its own. So you can scale [20:23] the amount of data you can evaluate. [20:26] And then additional Additionally, you [20:28] have a new space of agentic commerce [20:30] that we keep hearing about. The idea is [20:32] to build this end-to-end multistage [20:34] system where you have agentic retrieval, [20:38] but you also build it in a way that [20:40] agents can call your system as a tool. [20:43] And your structured inputs where you are [20:46] where you have constraints and budgets, [20:47] let's say if a user searches for I want [20:50] to buy HP cartridge and my budget is [20:53] XYZ, [20:54] we want to reason that and query [20:56] understanding can be evolved to capture [20:58] those details [20:59] and flag it as the flag the specific [21:03] entities and able to apply those [21:05] additional constraints into your system. [21:10] And I think if I were to say the three [21:13] takeaways from this talk that you should [21:15] talk about if you don't remember [21:16] anything that we discussed today, it's [21:19] no single model really wins in any [21:21] search system. [21:22] Exact product matching needs a pipeline, [21:24] not one embedding. [21:26] We should always fuse our results when [21:29] we're trying to generate relevant [21:30] candidates, [21:31] and we find and we we have a precise [21:33] re-ranker to get your top K results. [21:37] And personalization comes last. [21:40] And the third thing that I would say you [21:41] should take away from this talk is [21:43] you should always have humans in the [21:44] loop and LLMs only when you really need [21:47] them. And the right stage is constantly [21:50] evolve your system by compounding the [21:51] quality of time that you're spending on [21:54] it by using it either for HITL or for [21:57] reasoning. [21:59] Thank you, everyone. Uh I'm Sidharth [22:01] Athparia, and you can connect with me.