Lenskart Boosts E-Commerce Conversions 7% via Multimodal AI Agents

From Demo to Millions: The Observability Loop Behind Lenskart's Voice, Video & AR Agents with GeoIQ — Ai4

Ai4

Synchronized multimodal session replay and automated evaluation loops resolved early production failures across noisy real-world environments, enabling non-technical shoppers to complete complex prescription purchases online without visiting physical stores.

AI-assisted summary of From Demo to Millions: The Observability Loop Behind Lenskart's Voice, Video & AR Agents with GeoIQ — original source. Check the original for full context.

Source published: 2026-08-21

Detailed notes

## Background Published by Ai4 on YouTube on August 21, 2026, this talk features Tusheet Shrivastava, Founder and CTO of GeoIQ (a Lenskart company), discussing the engineering behind scaling multimodal retail agents. According to the page description, the session provides a practical blueprint for building an observability loop that combines traces, automated evaluations, human feedback, and production metrics to monitor voice, video, and AR agents in production.

## TL;DR Tusheet Shrivastava breaks down how Lenskart deployed and hardened an agentic voice, video, and AR shopping assistant to overcome the distinct frictions of prescription eyewear e-commerce. He explains the system's WebRTC multi-agent architecture and real-time bi-directional UI state synchronization, while candidly detailing severe early failure modes caused by transit noise, compound user intents, and dropped FAQ tool invocations. To resolve these P99 breakdowns, Lenskart built ScreenIQ and a three-layer observability loop featuring automated LLM judges, monotonic timeline replays, and regression testing, ultimately achieving a 7% lift in digital conversion rates across millions of users.

## Atmosphere & style The presentation carries a pragmatic, engineering-first tone, pivoting from an ambitious architectural walkthrough to an honest post-mortem of real-world production failures. Shrivastava speaks with candid authenticity about the sleepless nights and messy edge cases that traditional dashboard averages masked during early deployment. The delivery feels technical, grounded, and deeply empathetic to diverse end users, highlighting the hard-won lessons required to turn fragile AI demos into resilient production systems.

## Key takeaways - [00:04](https://www.youtube.com/watch?v=mFB21_l8rag&t=4s) Lenskart operates over 2,800 omnichannel retail stores and eye care centers across multiple geographies alongside its digital platform. - [01:31](https://www.youtube.com/watch?v=mFB21_l8rag&t=91s) Prescription eyewear e-commerce involves complex variables—including pupillary distance (PD), ocular powers, lens indices, and specialized coatings—that create significantly higher friction than standard retail checkout flows. - [01:31](https://www.youtube.com/watch?v=mFB21_l8rag&t=91s) Older demographics require the most complex lens configurations but struggle most with dense app interfaces, making interactive voice and video shopping assistants essential. - [04:42](https://www.youtube.com/watch?v=mFB21_l8rag&t=282s) Eyewear shoppers frequently describe their optical needs in colloquial, unstructured terms and non-technical language (such as describing photochromic lenses by their behavior in sunlight). - [06:00](https://www.youtube.com/watch?v=mFB21_l8rag&t=360s) The multimodal shopping assistant conducts end-to-end sales consultations by analyzing facial landmarks via camera feed, rendering AR try-ons, comparing lens tiers, and applying promotions directly to cart. - [09:03](https://www.youtube.com/watch?v=mFB21_l8rag&t=543s) The technical architecture relies on low-latency WebRTC media streams coordinated across specialized multi-agent systems with vector database search and RAG knowledge retrieval. - [09:03](https://www.youtube.com/watch?v=mFB21_l8rag&t=543s) Bi-directional UI state synchronization is critical to keep the conversational agent and user touch/scroll actions aligned in real time throughout the session. - [12:07](https://www.youtube.com/watch?v=mFB21_l8rag&t=727s) Lenskart developed ScreenIQ, an observability tool that reconstructs audio, video landmarks, prompt context, tool calls, and user UI gestures into a unified, monotonic timeline for replay. - [15:11](https://www.youtube.com/watch?v=mFB21_l8rag&t=911s) Initial production deployment revealed critical P99 failure modes, including metro/bus ambient noise degradation, uncalled FAQ tools causing hallucinations, and failure to handle compound intents. - [17:15](https://www.youtube.com/watch?v=mFB21_l8rag&t=1035s) A three-layer observability loop combines automated post-session LLM judges, ScreenIQ visual replay filtering, and trace-level regression testing to systematically harden agents. - [17:15](https://www.youtube.com/watch?v=mFB21_l8rag&t=1035s) Hardening the multimodal assistant produced a 7% increase in digital conversion rate, driven largely by older customers who previously abandoned digital checkouts for physical stores.

## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=mFB21_l8rag&t=4s) Introduction and Lenskart's Omnichannel Scale

Tusheet Shrivastava introduces his background of over 15 years building machine learning and data science systems. He co-founded GeoIQ, a spatial intelligence company that was acquired by Lenskart, and has since been leading the development of production-grade AI agents at Lenskart. The flagship project is an AI stylist and shopping assistant deployed at massive scale across India to assist millions of customers in finding, styling, and purchasing eyewear.

Lenskart operates as India's largest eyewear manufacturer and retailer, managing an omnichannel network of over 2,800 physical stores across multiple international geographies alongside its digital platforms. The scale encompasses physical eye care centers and digital customer journeys, requiring seamless continuity between online and offline shopping behaviors for a vast user base.

- **Speaker Background:** Tusheet Shrivastava (Founder & CTO, GeoIQ; AI Engineering Lead at Lenskart; 15 years in ML/data science). - **Lenskart Physical Footprint:** 2,800+ omnichannel retail stores and eye care centers across multiple geographies.

---

[01:31](https://www.youtube.com/watch?v=mFB21_l8rag&t=91s) The Friction of Prescription Eyewear vs. Traditional E-Commerce

Shopping for prescription eyewear presents operational and technical hurdles that do not exist in standard e-commerce. In traditional retail journeys, a shopper selects a stock keeping unit (SKU), picks a size, completes checkout, and waits for shipment. In contrast, prescription eyewear requires selecting a frame geometry, ensuring the correct pupillary distance (PD)—the physical measurement between the eyes, which causes severe vision issues if miscalculated—configuring independent left and right eye sphere/cylinder powers, selecting specialized lens packages, and deciding on optional coatings such as anti-glare, blue-light filtering, and scratch resistance.

```mermaid flowchart TD accTitle: Traditional vs Eyewear E-Commerce Flow subgraph Traditional["Traditional E-Commerce"] T1["Select SKU"] --> T2["Choose Size"] --> T3["Checkout & Ship"] end subgraph Eyewear["Prescription Eyewear E-Commerce"] E1["Select Frame & Shape"] --> E2["Compute Facial Landmarks & PD"] E2 --> E3["Input Left / Right Power"] E3 --> E4["Choose Lens Package & Coatings"] E4 --> E5["Cart & Checkout"] end ```

Because of these technical complexities, customers are flooded with lifestyle and clinical questions that a static web page or standard app interface cannot resolve. Consequently, a large portion of shoppers historically defaulted to visiting physical stores to ask optometrists and store staff questions like "does this look good on me?" or "what kind of lens package should I buy if I have these preferences?" Furthermore, Lenskart's customer demographic in India spans from 8-year-olds to 80-year-olds. The elderly demographic requires the most complex multifocal and high-index lenses, yet they struggle the most with dense digital application interfaces. They are far more comfortable speaking naturally about their optical needs, creating a strong imperative for an interactive voice, video, and augmented reality shopping assistant.

- **Prescription Variables:** Pupillary distance (PD), independent left/right ocular power, frame face fit, lens refractive index, and specialized coatings. - **Demographic Spread:** Users aged 8 to 80; older users have the highest optical package complexity but the lowest fluency with complex app checkout flows.

---

[04:42](https://www.youtube.com/watch?v=mFB21_l8rag&t=282s) Conversational Search and Natural Customer Language

Traditional search engines depend on structured keywords, but real-world eyewear shoppers describe their aesthetic and functional needs in colloquial, unstructured terms. Users frequently ask the AI stylist qualitative style questions such as "will it look like a boring corporate or would it look like a cool Gen Z?" or ask for a "classy minimal old money vintage look."

These expressive queries also cross linguistic boundaries. In Hindi and multilingual Indian contexts, customers articulate technical requirements using descriptive behavior rather than technical optical jargon. For instance, customers describe photochromic lenses by asking in Hindi for "a frame which turns darker when there is sunlight and it turns lighter when there is shade," entirely omitting the industry term "photochromic." Traditional search bars fail on these queries, whereas an agentic conversational assistant can interpret the underlying intent and surface the right catalog items.

- **Aesthetic Intent:** Queries centered around cultural archetypes ("cool Gen Z", "old money vintage", "boring corporate"). - **Unstructured Optical Inquiries:** Functional descriptions of transition lenses without knowing technical terms like "photochromic".

---

[06:00](https://www.youtube.com/watch?v=mFB21_l8rag&t=360s) Step-by-Step Multimodal Agent Session Walkthrough

A live interaction showcases the end-to-end capabilities of the AI stylist. The session begins with the agent prompting the user in their preferred language, referencing previous browsing preferences (such as square transparent TR90 frames), and offering new recommendations. When the user asks for glasses that "actually suit my face," the agent accesses the real-time camera feed, analyzes facial geometry, and highlights angular, geometric, and square frames to add facial definition.

```mermaid flowchart LR accTitle: Customer Journey Through AI Voice and AR Agent subgraph Discovery["1. Discovery & Fit"] A["Voice Greeting"] --> B["Face Landmark Analysis"] B --> C["AR Frame Try-On"] end subgraph Inquiry["2. Product FAQ"] C --> D["Query: Weight & Comfort"] D --> E["FAQ Tool: TR90 Material & Nose Pads"] end subgraph Optical["3. Prescription & Cart"] E --> F["Power Selection: Distance Vision"] F --> G["Package Compare: Blue Screen vs OWNDAYS"] G --> H["Apply BOGO & Add to Cart"] end ```

The user proceeds through a complete sales and optometry consultation: 1. **Facial Feature Styling:** The agent overlays the frames onto the user's face using augmented reality (AR) try-on while explaining why geometric shapes balance their facial features. 2. **Product Deep-Dive & FAQ:** The user asks if the frames are heavy for all-day wear. The agent identifies the active frame as part of the *Lenskart Air* collection, detailing its lightweight TR90 material, stainless steel temples, and adjustable nose pads. 3. **Power Configuration:** When the user indicates they cannot see far-off objects (myopia), the agent maps the symptom to the "with power" option. 4. **Lens Package Comparison:** The agent compares three lens tiers: *Anti-Glare Premium* (double-sided anti-glare, scratch resistance), *Lenskart Blue Screen* (digital eye strain protection), and *OWNDAYS Japan Blue Plus Eye Shield* (advanced screen protection, crack/smudge resistance, and night driving coating). 5. **Price Sensitivity & Checkout:** When the user notes that the OWNDAYS package is too expensive for their budget, the agent seamlessly pivots to recommend the more affordable Lenskart Blue Screen package, applies a "Buy One Get One" (BOGO) promotion, and adds the configured product directly to the cart.

---

[09:03](https://www.youtube.com/watch?v=mFB21_l8rag&t=543s) Multi-Agent Architecture and Synchronized UI State

The technical engine powering the assistant relies on low-latency WebRTC media rooms that stream real-time audio and video from the mobile application. Behind this media layer sits a coordinated multi-agent orchestration network where specialized agents seamlessly hand off control depending on conversational stage: product discovery, AR virtual try-on, lens package selection, add-on customization, and cart checkout.

```mermaid flowchart TD accTitle: Lenskart Multi-Agent Architecture and State Sync subgraph Client["Client Application Layer"] App["Lenskart Mobile App (WebRTC Room)"] AR["Real-Time AR Try-On & UI Screen"] end subgraph Orchestration["Agent Orchestration Network"] Router["Session Router / Noise Filter"] AgentDisc["Product Discovery Agent"] AgentAR["AR Try-On Agent"] AgentLens["Lens Package Selector Agent"] AgentCart["Cart & Checkout Agent"] end subgraph Retrieval["Data & Retrieval Layer"] VecDB["Vector DB (Semantic Catalog Search)"] RAG["FAQ / Warranty RAG Engine"] Profile["User Profile & 100+ Face Landmark Points"] end

App <--> |Media Streams| Router Router --> AgentDisc & AgentAR & AgentLens & AgentCart AgentDisc <--> VecDB AgentLens <--> RAG AgentAR <--> Profile AgentDisc & AgentAR & AgentLens & AgentCart --> |Bi-directional State Updates| AR ```

Underneath the agent network, semantic search is executed via vector databases for natural language catalog discovery, alongside Retrieval-Augmented Generation (RAG) engines that ingest extensive knowledge bases on warranties, returns, and optical technicalities. The agent constantly incorporates real-time user profile attributes and computer vision landmarks.

The fundamental engineering challenge is **bi-directional UI state synchronization**. The agent does not simply generate voice responses; it controls the client-side visual interface in real time, rendering frame carousels, toggling AR glasses on the user's face, and updating product detail cards. Simultaneously, the user can manually tap the screen, swipe back, or inspect alternate SKUs. The agent must maintain total awareness of screen state at every millisecond to gracefully react whether an interaction is voice-driven, touch-driven, or an interleaved combination of both.

- **Streaming Protocol:** WebRTC audio and video rooms operating within the Lenskart native mobile app. - **Agent Subsystems:** Product Discovery, AR Try-On, Lens Package Selector, Add-on Handler, and Cart Manager. - **Knowledge Retrieval:** Vector database semantic search combined with FAQ RAG systems.

---

[12:07](https://www.youtube.com/watch?v=mFB21_l8rag&t=727s) ScreenIQ: Monotonic Timeline and Multimodal Replay

To debug complex, multimodal interactions where speech, vision, tool invocations, and manual screen gestures collide, Lenskart developed a proprietary observability platform called **ScreenIQ**. Standard backend distributed traces and span logs fall short because they fail to capture the visual and temporal context experienced by the end user.

ScreenIQ provides an in-sync, monotonic timeline that reconstructs every session event in strict chronological order. For any interaction, engineering and QA teams can jump to a specific timestamp to inspect: - The exact prompt context and system instructions loaded into the model. - Incoming audio cleaned of ambient noise and the resulting transcription. - Vision pipeline outputs, including over 100 facial landmark features used for aesthetic recommendations. - Downstream tool calls (e.g., catalog search, FAQ retrieval, cart mutations) along with their respective latencies. - The visual UI screen state and user touch inputs occurring alongside voice commands.

This visual replay loop removes ambiguity from post-incident debugging, allowing engineers to verify whether an awkward response stemmed from an ASR error, a slow tool invocation, a model reasoning failure, or an unexpected manual tap by the user.

- **Facial Landmark Pipeline:** Evaluates 100+ facial feature points in real time to drive shape recommendations. - **ScreenIQ Monotonic Timeline:** Unifies audio traces, visual frames, tool calls, and user tap events into a single replayable timeline.

---

[15:11](https://www.youtube.com/watch?v=mFB21_l8rag&t=911s) Production Hardening: P99 Failures and the Eight Failure Modes

Tusheet reflects candidly on the initial deployment of the agent, describing the first launch as "terrible" due to the overwhelming number of failure modes encountered in the wild. While standard metrics and average latencies appeared acceptable during internal staging, real-world customer sessions exposed severe breakdowns at the P99 tail.

```mermaid flowchart TD accTitle: Post-Launch Failure Modes Breakdown subgraph Failures["Production Failure Modes"] F1["Acoustic Ambient Noise (Metros & Buses)"] F2["Misunderstood Intent & Query Drifts"] F3["Uncalled or Misdirected FAQ Tool Calls"] F4["Product Hallucinations"] F5["Mismatched UI vs Conversation States"] F6["High P99 Tool Latencies"] end ```

Intensive debugging sessions and sleepless nights led the team to isolate eight primary failure categories that crippled the early agent experience: 1. **FAQ Retrieval Gaps & Hallucinations:** The agent failed to invoke the RAG FAQ tool when asked detailed product or material questions, leading it to hallucinate frame specifications. 2. **Acoustic Noise Degradation:** Users frequently engaged with the voice assistant in noisy environments like public buses, trains, and metro stations. Cross-talk and ambient transit noise polluted speech recognition, causing intent classification to drift wildly. 3. **Erroneous Tool Invocations:** The model triggered incorrect backend tools or executed them with invalid argument payloads. 4. **Multiple Intent Failures:** When users combined multiple conversational threads in one breath—such as evaluating facial aesthetics while simultaneously questioning frame durability—the agent often answered one aspect while misinterpreting or dropping the other entirely.

- **Deployment Realities:** Average latencies masked critical P99 latency spikes and tool execution failures. - **Primary Failure Modes:** Metro/bus background acoustic noise, dropped FAQ tool calls resulting in hallucinations, intent drift on compound queries, and desynchronized tool execution.

---

[17:15](https://www.youtube.com/watch?v=mFB21_l8rag&t=1035s) The 3-Layer Observability Loop and Business Impact

To create a self-healing development pipeline, Lenskart implemented a three-layer observability loop that combines automated evaluation with deterministic root-cause analysis:

1. **Post-Session LLM Judges:** Following every completed session, automated LLM judges evaluate the conversation history, scoring retrieval accuracy, conversational coherence, tool invocation correctness, and whether recommended frames satisfied the customer's constraints. 2. **Log Filtering & Session Replay:** Sessions receiving low judge scores or negative user feedback are automatically isolated. Engineers use ScreenIQ to replay the session's monotonic timeline and pinpoint the exact moment of failure. 3. **Trace Remediation & Regression Testing:** Engineers inspect trace spans, patch prompt boundaries, refine noise-filtering thresholds, or fix tool-calling schemas. The corrected pipeline is re-run against historical queries to verify that the failure mode is resolved before pushing to production.

```mermaid flowchart LR accTitle: Three-Layer Observability Closed Loop A["Production Sessions (Millions of Users)"] --> B["Layer 1: Automated LLM Judges"] B --> C["Filter Low-Relevance & Faulty Turns"] C --> D["Layer 2: ScreenIQ Session Replay"] D --> E["Layer 3: Trace Debugging & Fixes"] E --> F["Regression Query Testing"] F --> A ```

This production hardening transformed the agent into an effective revenue driver used by millions of customers across India. The deployment has achieved a **7% increase in conversion rate** across Lenskart's digital ecosystem. The most dramatic gains have emerged among older demographics who previously abandoned digital checkouts to visit physical stores for basic reassurance on prescriptions, warranties, and frame fits, but can now complete their purchases independently through voice-guided AI shopping.

- **Automated LLM Judges:** Score session turns post-call on retrieval relevance, tool invocation accuracy, and conversational quality. - **Conversion Metric:** 7% overall increase in digital conversion rate across Lenskart. - **Primary Beneficiary Segment:** Elderly and non-tech-fluent users successfully converting via assisted voice and video shopping without needing store visits.

## Notable quotes > Building a compelling AI demo is easy; making it reliable for millions of customers is the real challenge. — Tusheet Shrivastava

> [01:31](https://www.youtube.com/watch?v=mFB21_l8rag&t=91s) does this look good on me?

> [01:31](https://www.youtube.com/watch?v=mFB21_l8rag&t=91s) what kind of lens package should I buy if I have these preferences?

> [04:42](https://www.youtube.com/watch?v=mFB21_l8rag&t=282s) will it look like a boring corporate or would it look like a cool Gen Z?

> [04:42](https://www.youtube.com/watch?v=mFB21_l8rag&t=282s) classy minimal old money vintage look

> [04:42](https://www.youtube.com/watch?v=mFB21_l8rag&t=282s) a frame which turns darker when there is sunlight and it turns lighter when there is shade

> [06:00](https://www.youtube.com/watch?v=mFB21_l8rag&t=360s) actually suit my face

> [15:11](https://www.youtube.com/watch?v=mFB21_l8rag&t=911s) terrible — Tusheet Shrivastava

## Mentioned Tusheet Shrivastava, GeoIQ, Lenskart, Ai4, Lenskart Air, Anti-Glare Premium, Lenskart Blue Screen, OWNDAYS Japan Blue Plus Eye Shield, ScreenIQ, WebRTC

## Video description Tusheet Shrivastava - Founder, CTO, GeoIQ - A Lenskart Company

Building a compelling AI demo is easy; making it reliable for millions of customers is the real challenge. I will share how we built an observability loop at Lenskart to capture real-world interactions, uncover failures, and continuously improve our voice, video, and AR agents. The talk will explore how we monitor latency, conversation quality, visual understanding, product recommendations, and virtual try-on as parts of one connected customer experience. Attendees will learn how we combine traces, automated evaluations, human feedback, and production metrics to detect issues that conventional dashboards miss. Join this session for a practical blueprint for taking multimodal agents from promising demos to dependable production systems.

[Source](https://www.youtube.com/watch?v=mFB21_l8rag)

中文简报

Lenskart借助多模态AI智能体将电商转化率提升7% 同步多模态会话回放与自动化评估闭环解决了嘈杂现实环境中的早期生产故障，使非技术型消费者无需前往实体店即可在线完成复杂的处方眼镜购买。 Tusheet Shrivastava详细剖析了Lenskart如何部署并强化具备智能体能力的语音、视频和AR导购助手，以克服处方眼镜电商领域的独特阻力。他阐释了该系统的WebRTC多智能体架构与实时双向UI状态同步，同时坦诚分享了由交通噪音、复合用户意图和FAQ工具调用遗漏所导致的早期严重失效模式。为解决这些P99级别的故障，Lenskart构建了ScreenIQ以及包含自动化LLM裁判、单调时间线回放和回归测试的三层可观测性闭环，最终在数百万用户中实现了数字渠道转化率7%的提升。除数字化平台外，Lenskart还在多个地区运营着超过2800家全渠道零售店和眼科护理中心。 (0:04) 处方眼镜电商涉及瞳距（PD）、屈光度、镜片折射率及特殊镀膜等复杂变量，相比标准零售结账流程存在显著更高的阻力。 (1:31) 年长客群往往需要最复杂的镜片配置，却最难适应密集的App界面，这使得交互式语音和视频导购助手变得不可或缺。 (1:31) 配镜消费者经常使用口语化、非结构化的措辞和非技术语言来描述自身的光学需求（例如通过在阳光下的表现来描述变色镜片）。 (4:42) 该多模态导购助手通过摄像头画面分析面部关键点、渲染AR试戴效果、对比不同档次镜片，并将优惠直接应用至购物车，从而完成端到端的销售咨询。 (6:00)

Reading summary

Lenskart deployed a multimodal voice, video, and augmented reality shopping assistant across its retail ecosystem to bridge the operational complexity of prescription eyewear and physical-store dependence. According to AI engineering lead Tusheet Shrivastava, resolving edge-case production failures through a synchronized multi-agent architecture and a closed-loop observability pipeline yielded a 7% increase in digital conversion. Eyewear Complexity and Colloquial Customer Queries (1:31) Prescription eyewear retail introduces clinical and geometric constraints absent in standard e-commerce, including pupillary distance measurements, independent left and right sphere and cylinder powers, lens refractive indices, and functional coatings. Shrivastava noted that while Lenskart serves customers aged 8 to 80 across more than 2,800 physical stores, older demographics require the most intricate optical configurations but struggle most with dense application checkout interfaces, historically forcing them to visit physical stores for guidance from optometrists. Furthermore, customer queries rarely conform to structured catalog keywords. Shoppers frequently describe style preferences through cultural archetypes like 'cool Gen Z' or 'old money vintage,' or articulate functional optical requirements through everyday descriptions. In Hindi and multilingual Indian settings, customers often request transition lenses by describing frames that darken in sunlight and lighten in shade without using the industry term 'photochromic.' Traditional keyword searches fail on these queries, requiring an agentic conversational interface capable of interpreting unstructured intent. Synchronizing Multi-Agent State Across Voice and Screen (9:03) Lenskart's shopping assistant runs over low-latency WebRTC audio and video streams embedded in its mobile application. The backend orchestration network routes sessions across specialized agent modules handling product discovery, augmented reality try-on, lens tier comparisons, and cart management. These agents query vector databases for semantic catalog matching and retrieval-augmented generation systems for warranty and FAQ data, while continuously tracking over 100 real-time facial landmark points to recommend frames tailored to facial geometry. A primary engineering challenge is maintaining bi-directional state synchronization between the conversation and the visual user interface. The agent dynamically renders product carousels, overlays AR frames, and modifies lens packages, while users can simultaneously tap the screen, swipe back, or inspect alternate items. To prevent desynchronization, the system continuously unifies visual screen states with voice inputs to handle interleaved touch and speech gestures seamlessly. Production Breakdowns at the P99 Latency Tail (15:11) Although internal staging benchmarks appeared acceptable, Shrivastava described the initial production deployment as 'terrible' due to extensive failures at the 99th percentile (P99). Field usage across Indian transit environments, such as public buses and metro stations, introduced severe background acoustic noise that degraded automatic speech recognition and caused conversational intent classification to drift. The team isolated eight core production failure modes. Beyond acoustic interference, the model frequently neglected to call internal FAQ retrieval tools during product inquiries, resulting in hallucinated frame specifications. The system also struggled with compound queries: when customers combined aesthetic questions with durability inquiries in a single breath, the agent routinely answered one component while dropping the other or executing backend tools with invalid payloads. Automated Evaluation and the ScreenIQ Diagnostic Loop (17:15) To systematically resolve production defects, Lenskart implemented a three-layer observability loop. In the first layer, automated post-session LLM judges evaluate completed transcripts for retrieval relevance, tool invocation accuracy, and conversational coherence. Flawed sessions are automatically flagged and transferred to ScreenIQ, a proprietary diagnostic platform that reconstructs an in-sync, monotonic timeline combining model prompts, cleaned audio, facial landmarks, tool latencies, and screen UI interactions. Engineers use this visual replay to isolate whether an error originated from speech recognition degradation, slow database queries, or model reasoning errors. After adjusting prompt boundaries, noise thresholds, or tool schemas, the pipeline is validated against historical regression suites. Shrivastava stated that this hardening process drove a 7% digital conversion rate increase, with the largest adoption gains occurring among older, less tech-fluent shoppers who can now complete optical checkouts independently. Lenskart's experience indicates that moving multimodal retail agents from proof-of-concept to production scale requires continuous, time-synchronized visual and audio telemetry rather than basic backend logging. Managing transit acoustic noise, interleaved screen gestures, and compound conversational queries remains a critical operational priority as the architecture expands across global omnichannel operations.

精炼阅读摘要

Lenskart 人工智能工程主管 Tusheet Shrivastava 指出，针对处方配镜流程高度复杂与用户表达非结构化的痛点，团队构建了基于 WebRTC 的多模态语音与 AR 导购智能体，并通过自研可观测平台 ScreenIQ 建立闭环调试机制，在解决长尾故障的同时使整体数字化转化率提升了 7%。处方配镜的高摩擦与自然语言诉求 (1:31) 处方眼镜电商面临远超常规快消零售的认知与配置门槛。传统网购只需选定 SKU 和尺码即可结算，而配镜则需要精确测定瞳距（PD）、独立配置左右眼屈光度与散光度数、权衡镜片折射率，并选择防蓝光或抗划伤涂层，其中瞳距的微小偏差即可引发佩戴不适。此外，Lenskart 在印度的消费群体横跨 8 岁至 80 岁。高龄用户往往需要装配最复杂的多焦点或高折射率镜片，却最不适应繁复密集的数字化结算界面，天然倾向于直接通过自然口语表达需求。消费者的真实诉求呈现出高度非结构化和多语言混杂的特征。用户常提出“老钱复古风”（old money vintage）或“酷炫 Z 世代”（cool Gen Z）等主观审美偏好；在印地语等语境下，顾客往往不会使用光致变色镜片（photochromic）等专业术语，而是描述为“太阳下变暗、阴凉处变亮”的功能特征。这种模糊且口语化的交互方式使传统的关键词搜索彻底失效，催生了对能够理解语境意图的会话式智能体的迫切需求。多智能体架构与端侧双向状态同步 (9:03) 为了支撑完整的售前与光学咨询，Lenskart 构建了基于 WebRTC 低延迟音视频流的多智能体协同网络。系统将会话请求路由至专门的子智能体模块，分别接管商品发现、AR 虚拟试戴、镜片套餐推荐、个性化配置及购物车结算。底层数据层由支持自然语言目录检索的向量数据库，以及处理质保和技术咨询的检索增强生成（RAG）知识库共同构成，并结合计算机视觉提取的 100 多个面部特征点实时生成几何推荐。 Shrivastava 指出，架构中最核心的工程挑战在于端侧双向 UI 状态的毫秒级同步。智能体不仅需要生成语音回应，还必须实时控制移动端的视觉呈现，包括滚动商品卡片、将 AR 镜架实时贴合至用户面部以及更新参数细节。由于用户在交互过程中可能随时手动滑动屏幕、回退或切换款式，系统必须精确保持智能体决策上下文与用户手势交互状态的完全一致，防止多模态交替输入引发会话脱节。长尾故障与 ScreenIQ 时间线回放 (12:07) Shrivastava 坦言智能体首次上线表现极差，内部测试阶段正常的平均延迟掩盖了严峻的 P99 尾部异常。团队总结了八类主要的生产失效模式：在公交和地铁等强噪环境中，背景杂音严重干扰语音识别导致意图漂移；智能体在被问及材质细节时未能触发 FAQ 工具，进而捏造镜架参数；后端工具调用失败或携带错误参数；以及当用户在一句话中并发提出审美与耐用度等多重意图时，智能体出现单侧遗漏或误判。为解决常规分布式链路追踪无法还原交互视效的短板，Lenskart 研发了专有可观测平台 ScreenIQ。该平台提供单调递增的统一时间线回放能力，将提示词上下文、降噪后的语音与转录文本、面部特征点视觉数据、工具调用耗时以及用户的实时触控手势完全对齐。工程人员得以在复盘时精准定位具体时间戳，厘清异常表现究竟源于语音识别失真、模型推理发散、底层接口超时还是用户的突发手动触控。三层可观测闭环与转化率成效 (17:15) 为消除长尾隐患，团队建立了三层自愈式研发闭环：第一层，在会话结束后由自动化 LLM 评估器对交互记录打分，审查知识检索相关性、工具调用准确率以及镜架推荐是否符合约束；第二层，系统自动筛选低分及负反馈会话，由工程师借助 ScreenIQ 单调时间线进行根因剖析；第三层，针对性优化提示词边界、修正降噪过滤阈值并重构工具调用模式，在历史回归查询集上验证通过后方可上线部署。该套工程化体系支撑了智能体在印度数百万用户规模下的稳定运行，并推动 Lenskart 整体数字化转化率提升了 7%。这一增长主要来源于以往因不擅长操作手机界面、担忧处方参数错误而必须前往线下 2,800 多家实体门店的老年用户，该群体通过语音交互与 AR 试戴成功在移动端独立完成了端到端的购镜流程。该案例表明，多模态智能体实现规模化商业落地的核心不仅在于对话生成，更在于应对现实环境声学噪声的韧性、双向 UI 状态的毫秒级同步以及基于全时序会话回放的调试闭环；但演讲并未讨论该方案跨出印度市场进入更多语言与复杂网络环境时的迁移成本与长期维护难度。

Detailed notes — zh

## Background 该视频由频道 Ai4 于 2026 年 8 月 21 日发布在 YouTube 平台上。视频记录了 GeoIQ 创始人兼 CTO、现任职于 Lenskart 的 Tusheet Shrivastava 带来的主题演讲，主要分享了团队如何构建一套贯穿语音、视频与 AR 试戴的多模态可观测性闭环，将 AI 智能体从概念 Demo 落地为服务数百万用户的可靠生产系统。 ## TL;DR 针对处方眼镜复杂的选品决策流程与印度多元的语言、年龄客群痛点，Lenskart 构建了一套集成 WebRTC 音视频流、多智能体网络和端控 UI 的全语音购物与 AR 试戴体系。上线初期，系统在真实场景中遭遇了高背景噪音干扰、复合意图误判及 FAQ 工具未触发导致模型幻觉等八大类核心故障。为此，团队自研了全链路可观测性平台 ScreenIQ，结合单调时序对齐的会话回放（Session Replay）与离线 LLM Judges 自动化评测建立三层排障闭环，成功攻克长尾 P99 缺陷，最终帮助非数字原生用户打通线上购买门槛，使全站转化率提升了 7%。 ## Atmosphere & style 演讲整体呈现出极其务实、硬核的技术复盘风格，主讲人以沉稳老练的口吻直面生产落地中的真实痛点，毫不掩饰初期系统表现不佳时的狼狈与挑战。随着从排查长尾 P99 故障、自研 ScreenIQ 可观测系统到斩获业务成效的层层剖析，叙事节奏从严峻的技术危机平稳过渡到解决复杂工程问题后的自信从容，给听众带来扎实且极富实战价值的工程洞见。 ## Key takeaways - [00:04](https://www.youtube.com/watch?v=mFB21_l8rag&t=4s) Lenskart 拥有超过 2,800 家线下实体门店，其研发的多模态 AI 造型师（AI Stylist）已部署至生产环境，服务印度本土数百万真实用户。 - [01:31](https://www.youtube.com/watch?v=mFB21_l8rag&t=91s) 处方眼镜电商决策链路极其复杂，需要测量校准双眼瞳距（PD）、录入差异化度数并挑选镜片镀膜套餐，静态 App 表单难以承载此类专业咨询需求。 - [01:31](https://www.youtube.com/watch?v=mFB21_l8rag&t=91s) 面对年龄跨度大、语言多元且不擅长复杂 UI 操作的印度老年客群，全语音购物交互提供了比传统打字搜索更自然的购买旅程。 - [04:42](https://www.youtube.com/watch?v=mFB21_l8rag&t=282s) 用户倾向使用“适合我脸型”、“在阳光下颜色变深”等生活化口语描述需求，传统关键词搜索框完全失效，必须依赖端到端的语音与多模态理解。 - [04:42](https://www.youtube.com/watch?v=mFB21_l8rag&t=282s) 系统能够毫秒级分析人脸 100 多项特征推荐合适镜框，结合用户日常用眼场景对比镜片套餐特性，并在加购时自适应调整预算与优惠核销。 - [09:03](https://www.youtube.com/watch?v=mFB21_l8rag&t=543s) 系统底层基于 WebRTC 低延迟音视频流，构建了涵盖选品发现、AR 试戴、镜片配置与购物车的专业多智能体网络（Multi-Agent Network），并辅以向量检索与 FAQ RAG 知识库。 - [09:03](https://www.youtube.com/watch?v=mFB21_l8rag&t=543s) 多模态交互核心架构难点在于 Agent 不仅进行语音应答，还在实时控制端侧 UI 状态，必须与用户的物理触控滑动实现毫秒级的双向同步。 - [12:07](https://www.youtube.com/watch?v=mFB21_l8rag&t=727s) 自研可观测性平台 ScreenIQ 具备“单调时序对齐”能力与会话完整回放功能，能够精确复原模型决策瞬间的多模态上下文、工具调用与端侧 UI 渲染流。 - [14:30](https://www.youtube.com/watch?v=mFB21_l8rag&t=870s) 传统 APM 的平均指标极易掩盖致命问题，多模态智能体系统的真正故障和用户体验断崖通常集中在长尾的 P99 极值场景中。 - [15:48](https://www.youtube.com/watch?v=mFB21_l8rag&t=948s) 生产环境常见故障主要集中在地铁/公交强背景噪音导致的意图误判、FAQ 工具调用缺失导致的信息幻觉，以及复合主客观意图下的错误动作执行。 - [17:15](https://www.youtube.com/watch?v=mFB21_l8rag&t=1035s) 团队构建了“LLM Judges 自动化评判筛选低质会话 → 链路 Trace 与 Replay 复现定位 → 历史样本回归重跑验证”的三层闭环治理体系。 - [17:15](https://www.youtube.com/watch?v=mFB21_l8rag&t=1035s) 经过可观测性闭环优化的多模态智能体系统为 Lenskart 带来了 7% 的全站转化率提升，显著打破了非数字原生老年用户的线上决策与信任门槛。 ## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=mFB21_l8rag&t=4s) 演讲开场：从 GeoIQ 到 Lenskart 的 AI 智能体探索主讲人 Tusheet Shrivastava 开场介绍了自己的从业背景。他在机器学习与数据科学系统领域深耕了 15 年，曾创立空间智能公司 GeoIQ，该公司随后被印度眼镜巨头 Lenskart 收购。加入 Lenskart 之后，Tusheet 带领团队专注于研发多种前沿的 AI 智能体（AI Agents）。本次分享的核心重点是一款已经推向生产环境、正服务于印度数百万真实用户的多模态 AI 智能体——AI 造型师（AI Stylist）。 Lenskart 作为印度本土最大的眼镜制造商与零售品牌，在多个国家和地区线下运营着超过 2,800 家实体门店，构建了一个庞大的全渠道（Omnichannel）生态体系。这个生态不仅涵盖密集的实体店、眼科护理中心（Eye Care Centers），还包含规模庞大的线上数字化购物旅程，每天都在承载数以百万计的线上线下消费互动。 - **演讲者背景**：Tusheet Shrivastava（GeoIQ 创始人兼 CTO，现任职于 Lenskart，拥有 15 年 ML/数据科学经验）。 - **公司规模**：Lenskart 拥有 2,800+ 线下实体门店，覆盖线上与线下全渠道生态。 --- ### [01:31](https://www.youtube.com/watch?v=mFB21_l8rag&t=91s) 处方眼镜电商的天然复杂性与印度多元化挑战 Tusheet 深入对比了传统电商与处方眼镜（Prescription Eyewear）购买链路的本质区别。在标准电商场景中，用户通常只需选定 SKU、挑选尺码、完成支付即可等待发货；但处方眼镜的决策流程极其复杂：用户首先要选定镜框设计与尺寸，接着必须测量并设定双眼瞳距（PD, Pupillary Distance，若设定偏差将直接导致佩戴时视觉眩晕甚至视力受损），随后要在纷繁复杂的镜片套餐（Lens Packages）与镜片镀膜（Coatings）中做抉择，最后还要分别准确录入左右眼不同的度数参数。面对如此多专业性极强的决策节点，传统的静态 App 界面或常规 UI 根本无法充分解答用户的疑问，这导致大量消费者依然选择前往线下实体店，以便向专业店员咨询“这款到底适不适合我的脸型？”或者“针对我的日常用眼习惯该选哪种镜片套餐？”。另一方面，Lenskart 面临着极具挑战性的印度本土市场特征。印度语言种类极其繁杂，客群年龄跨度极大，从 8 岁的孩童一直延伸到 80 岁的高龄老人。特别是老年客群，他们往往需要更复杂的镜片配置（如渐进多焦点镜片）且更容易存在高度数问题，但这部分人群并不擅长操作复杂的 App 界面与层级表单。相比于在屏幕上打字翻找，他们更习惯且更舒服于用自然口语把自己的视力困扰和需求直接“说出来”。为此，Lenskart 团队决定打造一条融合“语音交互、视觉理解与实时推荐”的全语音购物旅程（Voice Shopping Journey）。 ```mermaid flowchart TD accTitle: 传统电商与处方眼镜电商决策流程对比 subgraph Traditional["传统电商购买链路"] T1["挑选 SKU"] --> T2["选择基础尺码"] --> T3["下单支付并寄送"] end subgraph Prescription["处方眼镜决策链路"] P1["挑选镜框设计与尺寸"] --> P2["测量并校准瞳距 (PD)"] P2 --> P3["录入左右眼差异化处方度数"] P3 --> P4["挑选镜片功能套餐"] P4 --> P5["选择特殊镜片镀膜 (防蓝光/夜间驾驶等)"] P5 --> P6["确认加购与结算"] end ``` --- ### [04:42](https://www.youtube.com/watch?v=mFB21_l8rag&t=282s) 突破传统搜索框限制的自然语言与多模态诉求传统搜索框在面对用户真实而感性的表达时往往显得无能为力。Tusheet 列举了真实用户向 AI Agent 提出的典型诉求：有人会问“戴上这个会显得像个无聊的打工人（Boring Corporate），还是像个酷炫的 Z 世代（Cool Gen Z）？”，有人直接表达“我想要一种高级、极简的老钱复古风（Classy minimal old money vintage look）”。更有印度用户用印地语（Hindi）描述需求：“我想要一种在阳光下颜色变深、在阴凉处自动变浅的镜框”，整句话完全没有提及“变色镜片（Photochromic）”这一专业词汇。这些充满个性化、生活化描述的自然语言，是传统结构化搜索框和分类筛选器完全无法承接的。为了应对这些需求，AI Agent 必须具备端到端的语音与视觉理解能力。随后现场播放了一段实机交互录像： - **脸型识别与款式推荐**：用户进入语音会话后提出“我想找适合我脸型的眼镜”，系统实时捕捉用户面部特征并回应：“根据您的面部特征，具有棱角的镜框能勾勒面部轮廓并完美衬托您的脸型。我们有方形和几何形状可供选择，您喜欢经典低调款还是带有一点风格创意的款式？” - **材质与细节追问**：用户追问是否有更多颜色并质疑日常佩戴会不会太沉，Agent 立即解释：“您当前试戴的是 Lenskart Air 系列，采用超轻 TR90 材质搭配不锈钢镜腿，日常佩戴非常舒适”，并准确确认该镜框配有独立鼻托。 - **处方与镜片套餐比对**：当用户表示“我看不清远处的物体（近视），该选哪个？”时，Agent 协助其选定有度数配置（With Power），并横向对比了防眩光高级版（Anti-Glare Premium）、防蓝光屏障（Blue Screen）以及高端 Japan Blue Plus 三款镜片的抗刮、防蓝光与防污特性。 - **预算自适应与优惠核销**：当用户提出推荐的 Japan Blue Plus 太贵、能否推荐平价替代款时，Agent 敏锐地降级推荐了性价比极高的 Lenskart Blue Screen 镜片，并在加购完成后主动提示用户已解锁“买一送一”（Buy One Get One）的专属优惠码。 --- ### [09:03](https://www.youtube.com/watch?v=mFB21_l8rag&t=543s) 底层技术架构：WebRTC、多智能体网络与端控 UI Tusheet 剖析了支撑这套实时体验的底层多模态架构。系统的底层通信依托于集成在 Lenskart App 内的 WebRTC 房间，用于承载超低延迟的实时音视频媒体流（Real-Time Media Streams）。在此之上，系统构建了一个高效协作的“多智能体网络”（Multi-Agent Network），其中包括专门负责选品发现（Product Discovery）、AR 虚拟试戴（AR Try-On）、镜片套餐推荐（Lens Package Selector）、加购结算（Cart）等多个专业子 Agent，在用户会话的各个阶段中实现顺滑的状态切换。同时，结合基于向量数据库（Vector DB）的产品语义检索和用于处理售后保修等长尾知识的 FAQ RAG 系统，配合用户画像数据完成精准推荐。在每一个单轮交互（Turn）中，流程包含音频降噪过滤、意图理解、模型决策、工具调用（Tool Call）、数据检索以及客户端屏幕界面的动态渲染。这里最复杂的核心难点在于：**Agent 不仅是在实时与用户语音对话，更在实时控制并生成客户端的 UI 状态，并且还需要与用户的物理触控操作实现双向同步**。当用户在屏幕上主动滑动返回或点击查看某个商品详情时，AI Agent 必须实时感知这一 UI 状态的变化并立刻调整自己的对话上下文。 ```mermaid flowchart LR accTitle: Lenskart 语音与 AR 智能体底层系统架构 subgraph Client["客户端交互层 (Lenskart App)"] RTC["WebRTC 音视频流"] UI["动态 UI 渲染 & AR 试戴"] end subgraph MultiAgent["多智能体网络 (Multi-Agent Network)"] Discovery["商品发现 Agent"] ARTryon["AR 试戴 Agent"] LensPkg["镜片套餐配置 Agent"] CartAgent["购物车/加购 Agent"] end subgraph Knowledge["数据与知识检索层"] VecDB["向量数据库 (语义搜索)"] RAG["FAQ / 保修 RAG 知识库"] Profile["用户特征与画像库"] end RTC <--> MultiAgent UI <--> MultiAgent MultiAgent <--> Knowledge ``` --- ### [12:07](https://www.youtube.com/watch?v=mFB21_l8rag&t=727s) ScreenIQ：面向端控 UI 与多模态会话的可观测性系统由于 UI 状态同时被 AI Agent 和真实用户两端并发控制，传统仅依赖 Trace、Span 和固定日志的 APM（应用性能监控）体系完全无法定位复杂的多模态异常。为了解决这一痛点，团队自研了名为 **ScreenIQ** 的全链路可观测性平台。 ScreenIQ 提供了一套具备“单调时序对齐”（In-sync Monotonic Timeline View）的会话回放能力。它完整按时间戳记录了一通交互中所发生的全部多模态事件流（包括音频输入、降噪前后对比、模型推理时上下文、工具调用触发、AR 渲染动作及客户端触控事件）。工程师可以随时在时间轴上任意回溯，精准重现“模型在做出某一步决策时，其上下文到底看到了什么屏幕状态和用户输入”，从而快速复现与定位异常行为。此外，ScreenIQ 结合了自动化评测大模型（LLM Judges）对模型的每次动作合理性进行打分，并全面监控首次响应时间（FRT, First Response Time）等核心系统指标。在现场演示 ScreenIQ 对上述案例的追踪详情中，展示了当用户提出“推荐适合我脸型的眼镜”时，系统是如何在毫秒级内触发视觉检测、从人脸提取超过 100 项细微特征，再联动推荐算法管道输出特定方形棱角镜框，随后又如何无缝路由到 FAQ 工具调用来回答镜架重量与舒适度问题的完整调用轨迹。 - **ScreenIQ 核心能力**： - **单调时序对齐（In-sync Monotonic Timeline View）**：精确复原每个决策瞬间的多模态上下文与 UI 真实状态。 - **会话完整回放（Session Replay）**：支持音频、视觉渲染与用户屏幕操作的同步重现。 - **LLM Judges 自动化评估**：对智能体工具调用及回复质量进行自动化判定。 - **视觉推荐流水线**：实时分析人脸 100+ 项特征，精准匹配镜框风格与几何结构。 ### [14:30](https://www.youtube.com/watch?v=mFB21_l8rag&t=870s) 会话回放与传统可观测性指标的局限在实际展示中，用户正在通过语音与智能体互动试戴 Lenskart Air 系列轻量眼镜，并询问诸如“是否有鼻托”等具体产品细节，智能体确认了该镜框配有鼻托并适合日常佩戴。系统能够自动标记并深入到任何一次用户会话中，调出关键标记点并重现用户经历的完整会话过程。这种会话重放（Session Replay）能力，让团队能够直观地检查用户体验是否流畅，或者在交互链条中是否存在未被察觉的问题。虽然团队跟踪了许多传统的系统指标——例如首字响应延迟（First Response Time）、用户的点击操作、智能体发起的工具调用（Tool Calls）以及工具调用的延迟等，但 Tusheet Shrivastava 指出，在传统可观测性中，真正的系统缺陷往往隐藏在 P99 极值中，而非默认指标或平均值里。在首次上线时，由于智能体存在大量的潜在故障点，团队并没有做好充分准备，导致初期表现“非常糟糕（terrible）”。 - **会话完整回放**：系统支持自动化提取交互过程中的关键标记（Key Markers），完整还原多模态交互全貌。 - **P99 极值排查**：平均延迟往往会掩盖偶发但致命的故障，关键故障大多发生在长尾的 P99 指标中。 - **基础指标监控**：全面跟踪包括工具调用耗时、用户交互点击及端到端首包响应在内的传统 APM 数据。 --- ### [15:48](https://www.youtube.com/watch?v=mFB21_l8rag&t=948s) 生产环境中的八大核心故障点经历了一系列熬夜奋战、系统调试以及构建回放功能后，团队成功定位了导致智能体表现不佳的八大核心故障领域。由于在实际落地场景中，FAQ 工具未能被正确调用，智能体频繁出现幻觉（Hallucination）。此外，真实环境中的噪声问题尤为突出——许多用户是在地铁或公交车等嘈杂环境中与语音智能体交互，环境杂音极易导致语音理解出现严重偏差。除了环境干扰外，意图理解失误（Misunderstood Intent）和工具调用错误（Wrong Tool Calls）也屡见不鲜。例如，用户本意是询问“这款眼镜适不适合我的脸型”，智能体却执行了完全不相关的操作。复杂的复合意图失败，直接导致了初版系统体验的崩塌。 ```mermaid flowchart TD accTitle: Multi-modal Agent Failure Modes subgraph InputFailures["输入与环境层"] A["高噪声环境 (地铁/公交)"] --> B["语音识别与意图误判"] end subgraph AgentFailures["智能体决策层"] B --> C{"意图解析与分发"} C -->|"多意图混淆"| D["执行错误动作"] C -->|"FAQ 工具未触发"| E["模型产生幻觉"] C -->|"选择错误 API"| F["错误工具调用"] end subgraph Impact["用户体验受损"] D --> G["推荐/试戴完全不匹配"] E --> G F --> G end ``` - **高噪音环境干扰**：地铁、公交车等强背景噪音极易误导语音模型，造成意图识别偏差。 - **工具调用失败与幻觉**：FAQ 检索工具未被正确触发时，模型会退化并编造产品信息。 - **复合意图失效**：当用户提出主观评价（如“我戴着好看吗”）与客观参数并存的复合问题时，智能体会产生行为偏离。 --- ### [17:15](https://www.youtube.com/watch?v=mFB21_l8rag&t=1035s) 三层可观测性闭环与 7% 转化率提升为了根治这些问题，团队建立了一套三层可观测性循环体系。首先，在会话结束后由 LLM 评判机制（LLM Judges）自动运行，筛选并找出检索质量低劣或与用户查询不匹配的异常会话。随后，团队通过缩小范围并调取详细日志（Logs）与追踪链路（Traces）定位具体症结，利用会话重放功能重现问题并修复底层逻辑，最后在相同输入样本上重新验证，确保修复有效。目前，这套经过严密可观测性闭环打磨的智能体系统已被数百万用户使用，并直接为 Lenskart 带来了全站转化率（Conversion Rate）7% 的提升。受益最明显的是不熟悉 App 操作的老年用户群体——以往他们为了弄清产品参数或保修政策等微小疑问，必须亲自前往线下实体门店咨询才能决定是否购买；如今，这套多模态智能体已经能够非常出色地解答他们的各种售前疑问，促成线上购买转化。 - **LLM 自动化评判（LLM Judges）**：离线扫描全量会话，自动捕获检索匹配度差或回答有偏差的低质交互。 - **会话回放与链路修复验证**：结合 Traces 与 Replay 功能重现问题，并在修复后通过重跑历史 Query 确认优化结果。 - **转化率显著增长**：智能体在生产环境中实现了 7% 的转化率提升，有效打通了老年等非数字原生用户在线上购买眼镜时的信任门槛。 ## Notable quotes > [04:42](https://www.youtube.com/watch?v=mFB21_l8rag&t=282s) 戴上这个会显得像个无聊的打工人（Boring Corporate），还是像个酷炫的 Z 世代（Cool Gen Z）？ — 用户 > [04:42](https://www.youtube.com/watch?v=mFB21_l8rag&t=282s) 我想要一种高级、极简的老钱复古风（Classy minimal old money vintage look）。 — 用户 > [04:42](https://www.youtube.com/watch?v=mFB21_l8rag&t=282s) 我想要一种在阳光下颜色变深、在阴凉处自动变浅的镜框 — 用户 > [04:42](https://www.youtube.com/watch?v=mFB21_l8rag&t=282s) 根据您的面部特征，具有棱角的镜框能勾勒面部轮廓并完美衬托您的脸型。我们有方形和几何形状可供选择，您喜欢经典低调款还是带有一点风格创意的款式？ — AI Stylist > [04:42](https://www.youtube.com/watch?v=mFB21_l8rag&t=282s) 您当前试戴的是 Lenskart Air 系列，采用超轻 TR90 材质搭配不锈钢镜腿，日常佩戴非常舒适 — AI Stylist > [04:42](https://www.youtube.com/watch?v=mFB21_l8rag&t=282s) 我看不清远处的物体（近视），该选哪个？ — 用户 > [09:03](https://www.youtube.com/watch?v=mFB21_l8rag&t=543s) Agent 不仅是在实时与用户语音对话，更在实时控制并生成客户端的 UI 状态，并且还需要与用户的物理触控操作实现双向同步。 — Tusheet Shrivastava > [14:30](https://www.youtube.com/watch?v=mFB21_l8rag&t=870s) 在传统可观测性中，真正的系统缺陷往往隐藏在 P99 极值中，而非默认指标或平均值里。 — Tusheet Shrivastava ## Mentioned Tusheet Shrivastava, GeoIQ, Lenskart, AI Stylist, ScreenIQ, WebRTC, Lenskart Air, Japan Blue Plus, TR90 ## Video description Tusheet Shrivastava - Founder, CTO, GeoIQ - A Lenskart Company Building a compelling AI demo is easy; making it reliable for millions of customers is the real challenge. I will share how we built an observability loop at Lenskart to capture real-world interactions, uncover failures, and continuously improve our voice, video, and AR agents. The talk will explore how we monitor latency, conversation quality, visual understanding, product recommendations, and virtual try-on as parts of one connected customer experience. Attendees will learn how we combine traces, automated evaluations, human feedback, and production metrics to detect issues that conventional dashboards miss. Join this session for a practical blueprint for taking multimodal agents from promising demos to dependable production systems. [Source](https://www.youtube.com/watch?v=mFB21_l8rag)

Source quotes

[4:42] 用户: 戴上这个会显得像个无聊的打工人（Boring Corporate），还是像个酷炫的 Z 世代（Cool Gen Z）？ [4:42] 用户: 我想要一种高级、极简的老钱复古风（Classy minimal old money vintage look）。 [4:42] 用户: 我想要一种在阳光下颜色变深、在阴凉处自动变浅的镜框 [4:42] AI Stylist: 根据您的面部特征，具有棱角的镜框能勾勒面部轮廓并完美衬托您的脸型。我们有方形和几何形状可供选择，您喜欢经典低调款还是带有一点风格创意的款式？ [4:42] AI Stylist: 您当前试戴的是 Lenskart Air 系列，采用超轻 TR90 材质搭配不锈钢镜腿，日常佩戴非常舒适 [4:42] 用户: 我看不清远处的物体（近视），该选哪个？ [9:03] Tusheet Shrivastava: Agent 不仅是在实时与用户语音对话，更在实时控制并生成客户端的 UI 状态，并且还需要与用户的物理触控操作实现双向同步。 [14:30] Tusheet Shrivastava: 在传统可观测性中，真正的系统缺陷往往隐藏在 P99 极值中，而非默认指标或平均值里。

Source quotes

Tusheet Shrivastava: Building a compelling AI demo is easy; making it reliable for millions of customers is the real challenge. [1:31] does this look good on me? [1:31] what kind of lens package should I buy if I have these preferences? [4:42] will it look like a boring corporate or would it look like a cool Gen Z? [4:42] classy minimal old money vintage look [4:42] a frame which turns darker when there is sunlight and it turns lighter when there is shade [6:00] actually suit my face [15:11] Tusheet Shrivastava: terrible

Original transcript

[0:04] Hello. Hi. Hi. Good afternoon everyone. I am Tushit. Uh I've been working for 15 years [0:12] building machine learning systems, data science systems and uh I also built a company called GOIQ. [0:19] Uh it's a spatial intelligence company which got uh acquired by Lenskart. [snorts] Um and [0:26] uh I've been since then working with lens cart in building um different types of AI agents. So today [0:32] I'll present a a very interesting uh agent that we have uh developed and is being used by millions of [0:39] people in India. So uh it's a it's a it's a AI stylist which helps you uh select glasses. So [0:51] first just a brief about lens card. So, Lenskart is the largest uh eyewear manufacturer and brand [0:58] in India. They operate like almost close to 2,800 plus stores across uh different geographies and [1:08] all it's it's it's a whole omni channel ecosystem where there are digital journeys that we have to [1:14] track. There are iClare I care centers and uh there are millions of people who shop [1:22] uh at the stores and on and online. So [clears throat] uh since we're talking about uh eyewear [1:31] uh I just wanted to highlight this that uh there there's a big difference in traditional e-commerce [1:39] and how people shop there and how you have to shop when you want to buy prescription eyewear. [1:46] So in traditional buying journeys, it's usually you pick a SKU, select a size and then you pay for [1:53] it and that particular SKU and uh gets uh shipped to you. But in prescription eyewear uh first you [2:02] have to select the design and then there are sizes inside uh the I frames also the distance between [2:10] your eyes the it's called the PD distance that needs to be set otherwise uh uh there are issues [2:17] in the vision uh then then there are a lot of lens packages and then there are lot of lens coatings [2:25] that you want to apply or not apply Uh and then you have to input your power and your left eye [2:33] power might be different from your right eye power and all of this has to be done online. And there [2:39] are lots of questions which people have in this uh journey and just the an app or a UI is not [2:49] able to do justice to the kind of questions that people have. Uh so uh there are a lot of people [2:58] uh go and buy them in stores because in the store you can ask a lot of questions to the staff like [3:05] is it does this look good on me uh like what kind of lens package should I buy if I have these [3:11] these preferences uh and all these uh different questions. Um then there is a second problem also [3:21] uh with lens card uh that their customer base uh a large chunk of that customer base in India. [3:33] Hello check. Yeah. Yeah. So um and India as you might know is like very diverse. [3:42] people speak lot of different languages uh and especially the target group which [3:49] uh lens card serves is uh like from 8 year olds to 80 year olds and even these 80 year olds need more [3:58] uh different types of lens packages and options and they are more prone to having [4:03] u some power in their lenses. So that just complicates uh the thing because these people [4:11] are also not very [snorts] uh fluent in using apps. uh oh they are sorry they are much more [4:20] comfortable in speaking and talking about their problems and issues and so that's why we built [4:26] like a voice shopping journey uh uh which can talk to you which can see you and it can recommend [4:35] uh eyelasses and or sunglasses and it can help answer a lot of questions. [4:42] Yeah. So [clears throat] this is the kind of question that people have [snorts] uh [4:48] on the app. Uh like these are questions which it's very hard to put inside a search box. Uh [4:56] these are things like uh like will will it look like a boring corporate or would it look like a [5:02] cool genzee? Uh like I want I want a classy minimal old money vage look. So people like [5:11] traditionally don't type this inside a search box. These these are some of the examples that I've put [5:17] uh of people asking our AI agent uh the kind of things that uh they want to look for. [snorts] Uh [5:24] then the first one is in Hindi. It says like uh I want a frame which turns darker when there is [5:32] sunlight and it turns white like lighter when there is shade. uh without particularly using [5:39] the keyword photochromatic. Uh so yeah there are various different kinds of questions that people [5:46] have which uh traditional UIs are not able to cater to well. So I'll just give you kind [5:54] of like a session of how this looks like. Welcome back. Please select a language to [6:01] continue your voice shopping experience. Want to continue exploring more square transparent TR90 [6:06] eyelasses or try something different today? Uh, I want some eyelasses that actually suit my face. [6:19] Based on your facial features, these angular frames will add definition and [6:22] complement your face perfectly. We have options in square and geometric shapes. [6:25] Would you prefer a classic understated look or something with a subtle style twist? Um, [6:31] I like these. Uh, are there more options? [6:41] Sure. Here are some more options. We have square frames in black and gunmetal and [6:44] also a rectangle half frame design. Do any of these interest you? Now, these look nice. Uh, [6:50] are they heavy? I have to wear them daily. So, um, will they uh be comfortable? [7:01] The ones you are trying on now are from Lensard Air. Made with lightweight TR90 material and [7:05] stainless steel temples, so they'll be comfortable for daily wear. Is there anything else you'd like [7:08] to know? Is there a nose pad in them? Yes, these frames have nose pads for a comfortable [7:14] fit. Anything else you'd like to know? Uh, I like this. Does it come with power? Yes, [7:20] these frames support power lenses. You can choose your prescription later in [7:23] the cart. Shall we look at other options first? No, I want to add this to cart. [7:32] Great choice. To add this to your cart, please select your power type first. The [7:36] options are with power, zero power for blue light block, progressive, your frame only. [7:40] Which one would you like? So, I cannot see far off objects. Which one should I choose? [7:48] You'll need the width power option. Then, should I confirm that for you? Yes, please confirm. [7:55] Great. I've confirmed with power. Now you can choose your lens package. The [7:58] options are anti-glare premium lens card blue screen or on days [8:02] Japan blue. Which one would you prefer? Can you compare them? [8:07] Sure. The anti-glare premium has double-sided anti-lare and is scratch resistant. The lens [8:11] card blue screen adds screen protection and minimizes eye strain. And the ones Japan [8:15] Blue Plus has advanced screen protection plus crack and smudge resistance. Which one sounds [8:18] best for you? So I want uh something for night driving as well as for uh screen protection. [8:28] For both night driving and screen protection, the Onday Japan Blue Plus Eye Shield is a great [8:32] choice as it has a night drive coating and advanced screen protection. Should I confirm [8:35] that for you? Uh, it's too expensive. Can I can you suggest a cheaper option? [8:43] In that case, the lens card blue screen is a good [8:45] option as it offers screen protection and minimizes eye strain and it's [8:48] more affordable. Should I confirm that one? Yes, let's go with that. [8:58] I've added that to your cart. You can unlock buy one get one on frames with code. Hi. [9:03] So, [snorts] yeah, what you just saw was uh the AI um not only uh recommending which frames suit the [9:13] face. It's able to see the user. It's able to put the frames on top of the user uh give feedback to [9:20] the user uh in real time uh answer their queries about product descriptions. Help them choose power [9:29] uh lens package options, cheaper options. It talks about offers and discounts whatever is [9:36] available and best applicable uh uh for the user. So uh quickly this is the engine that powers it. [9:46] uh it's based on uh web RTC uh rooms uh on the lens cut app the real-time [9:55] uh media uh streams and there is a uh multi- aent network uh uh so product discovery AR tryon which [10:05] you saw the add-on the lens package selector and the cart these are all different agents [10:10] switching uh and uh when when the interaction is happening with with the with with the user. Then [10:17] uh there's uh vector DB based semantic searching uh for products, FAQ, rag based systems for uh [10:26] answering uh detailed questions about warranties and whatnot. And uh then there is a lot of data [10:33] around the user profile that we get uh when the AI is actually recommending uh glasses. [10:42] So this is uh one one particular turn what happens like when the user speaks [10:49] uh there is noise and everything that has to be uh accounted for after cleaning the noise the agent [10:57] uh interprets that the model acts on it then it maybe does a tool call if required then the [11:03] tool calls uh retrieval runs then the screen also is updating. So what you saw in the demo [11:11] uh the person was just speaking uh and the screen was getting controlled by the agent in real time. [11:21] So th that is what brings us to the key problem uh or the the most difficult part of uh such an [11:31] agent because it's not only just in real time interacting with the user. It's also has to [11:38] uh generate the screen. It has to know what's on the screen always. It also has to react to [11:44] the user's tab. The user can also start using the screen. they can do back or they can do go [11:50] to a detail of any product if they want to and the AI agent has to react to that. So the UI state is [11:58] being controlled by the agent as well as the user uh which is uh which is what brings us to like [12:07] how do we do observability for such an agent. So uh instead of the normal obs like apart from the [12:14] normal observability and traces and spans that we have to do for all AI agents here we had to also [12:23] and also have uh the ability to replay sessions so as to know what what was the experience uh for [12:34] the user. Uh then we have judges uh which help us decide whether the model what what it acted [12:42] on was it correct or not and all the traditional uh metrics and FRT and all the metrics needed for [12:51] um monitoring whether the systems and the codes are working correctly or not. So uh this is screen [12:58] IQ what we built to particularly handle for this. Uh this is an insync uh monotonic timeline view of [13:08] all the events that happened in this conversation uh uh time stamp so that anytime we can go back in [13:17] the uh timeline and see what was the model's context at that time, what decision it did [13:23] uh it took, what kind of events had triggered that uh so that we can debug any issue that happens. [13:32] So yeah, so this is just that session uh I've put together on screen IQ also. So [13:40] I want some eyelasses that actually suit my face. Yeah. So here he asked for the [13:46] eyeglasses to s that will suit his face and depending on based on your facial features, [13:52] these angular frames will add definition. So these are the eyeglasses that the AI returned and it saw [13:59] his face first and then bases the face. Uh it ran our recommendation pipelines which look at like [14:06] 100 plus features across the face uh to recommend which eyeglasses would [snorts] work best and [14:13] uh then uh does it weigh less or not? These are like FAQ tool calls that were called. [14:19] Uh are they heavy? I have to wear them daily. So um will they uh be comfortable? [14:30] Yeah. So here the ones you are trying on now are from Lens Cart Air made with lightweight [14:34] phone calls here to get the details. So they'll be comfortable for daily wear. Is there anything else [14:38] you'd like to know? Is there a nose pad in them? Hello. Check. Yes, these frames have nose pads [14:43] for a comfortable fit. Anything else you Yeah. So uh as you can see we can go back into any session [14:50] look at the important key markers what happened in that session and all this happens automatically. [14:56] This is what helps us uh see whether uh the session the experience that the user had we [15:03] can replay that entire experience to see uh how uh whether there were any issues in this or not. [15:11] So and these are the traditional metrics that we have to track uh uh the the first response time [15:17] all the user taps all the agent tool calls that were done the tool call latencies and everything [15:22] this is traditional observability uh and uh of course every metric that you see in traditional [15:29] observability the real faults you see are at P99s and not at the defaults or averages and so our [15:38] first launch was like terrible because the agent had so many failure points. Uh and uh we were just [15:48] not like prepared I would say. Uh but uh we there were after a lot of sleepless nights and uh lot [15:57] of uh debugging then building all these tradition all these replay functionalities we were able to [16:05] identify the core eight areas where the agent was not performing well. Uh so there was a lot [16:13] of hallucination happening because the FAQ FAQ tool call was not getting properly called. These [16:20] are all the kind of different uh failure points uh in the system. Uh yeah, so there were a lot [16:29] of places where there was misunderstood intent, lot of noise based issues like people using this [16:36] within like a metro or a bus where there's a lot of noise coming from the side also which [16:42] makes it even harder for these voice agents to perform uh and get misleaded. Uh a lot of [16:52] uh wrong tool calls were called. Uh there were multiple intent failures like the person was [16:58] asking whether this will look good on me and the agent does something else altogether. Uh [17:03] so these were like some of the major ones which I wrote down. So uh this is uh the whole three [17:15] uh layers of observability that I talked about. Uh so first is the NLM judges are running. Uh [17:24] like suppose there's an issue that happened like B showed me wrong frames or we got some kind of [17:31] uh feedback around this. Uh so uh the judges first run they filter out what uh what were [17:40] the uh particular queries where there were uh very uh did not where the retrieval results were not as [17:49] uh good as compared to uh the query that was asked and of course this is done through LLM [17:56] judges running after the session. Uh then uh we narrow down the results, look at the logs, [18:03] see where what kind of issues were happening. We can replay the sessions once we identify where the [18:10] issues are and then we can uh look at again the traces, see what the issues were and then fix and [18:19] prove it that that uh we are able to run the same queries again and this works much better. Then [18:29] so yeah quickly coming to the kind of changes that we have seen this is an agent that is [18:34] being used by millions of people now uh and it has already uh resulted in a 7% increase [18:41] in conversion rate across lensard. So primarily uh these are the older people uh who are not [18:50] very used to the lens card application and always had to go to the store to [18:55] identify or for any small questions or doubts that they had about the product the warranty and [19:01] before buying or before making that commitment to the particular glasses. Uh these are being [19:07] handled by the model pretty well now. Yeah, that's pretty [snorts] much it. Thank you so