Mercari Drops AI Label and Cuts Verbose Prompts to Lift Seller Volume
Ai4
Mercari increased marketplace conversions and listing volume by treating generative AI as invisible utility rather than a conversational chatbot, proving that concise factual outputs and removing AI branding reduce user friction.
## Background
Published on YouTube by Ai4 on August 27, 2025, this talk features Max Frenzel, Head of Applied AI Research at Mercari. According to the description, Frenzel details the development of Mercari's multimodal AI Listing feature for its 23 million monthly active users, sharing practical insights on system design, prompt engineering, UX choices, and tangible business results beyond standard chatbots.
## TL;DR
Max Frenzel explains how Mercari tackled the intimidating 'blank page' problem in C2C commerce by creating a multimodal feature that turns item photos into complete marketplace listings in seconds. Rejecting chatbot interfaces in favor of seamless, near-invisible UX, Mercari integrated GPT-4o mini alongside an intentional category-confirmation step to maintain grounding and user trust. Post-launch iterations demonstrated that short, fact-dense, bulleted prompts drastically outperformed verbose descriptions by cutting down user edit rates and driving higher sales conversion. Furthermore, stripping 'AI' branding from the user interface bypassed consumer AI fatigue and boosted feature adoption across the platform's massive listing volume.
## Atmosphere & style
The presentation carries a crisp, pragmatic, and engineering-driven tone, balancing hard marketplace metrics with nuanced product psychology. Frenzel presents Mercari's journey with transparent candor, openly sharing launch-day anxieties, edge-case red-teaming anecdotes, and counterintuitive testing discoveries. It feels like an authoritative masterclass in production GenAI, cutting through typical tech hype to focus on practical utility and user trust.
## Key takeaways
- [00:04](https://www.youtube.com/watch?v=Ej-Vnu5RxjA&t=4s) Mercari handles immense scale in Japan with 23 million monthly active users and an average of 850 new listings created per minute, making listing friction a massive operational challenge.
- [03:07](https://www.youtube.com/watch?v=Ej-Vnu5RxjA&t=187s) Activating the long tail of 'never-listers' and light sellers represents the highest-leverage growth opportunity for C2C marketplaces compared to power sellers.
- [05:02](https://www.youtube.com/watch?v=Ej-Vnu5RxjA&t=302s) Chatbots were deliberately avoided for listing creation because multi-turn conversational interfaces replace one form of friction with another rather than removing it.
- [05:02](https://www.youtube.com/watch?v=Ej-Vnu5RxjA&t=302s) The core product philosophy was that the best AI implementation doesn't look like AI to the user, prioritizing seamless UX over conversational novelty.
- [06:45](https://www.youtube.com/watch?v=Ej-Vnu5RxjA&t=405s) Mercari architected the system betting on future model improvements, transitioning from GPT-4o to GPT-4o mini to enable cost-effective multi-image inputs and lower latency.
- [06:45](https://www.youtube.com/watch?v=Ej-Vnu5RxjA&t=405s) Introducing an intermediate category confirmation step grounded the multimodal LLM against cluttered backgrounds and built user trust through agency.
- [10:55](https://www.youtube.com/watch?v=Ej-Vnu5RxjA&t=655s) Extensive dogfooding with around 100 employees and auxiliary LLM evaluation tools helped surface hallucinations, safety edge cases, and policy compliance gaps.
- [14:03](https://www.youtube.com/watch?v=Ej-Vnu5RxjA&t=843s) Overly polished and corporate listing copy made peer-to-peer buyers suspicious, requiring prompts to be tuned toward an authentic, human tone.
- [17:04](https://www.youtube.com/watch?v=Ej-Vnu5RxjA&t=1024s) Transitioning from verbose, descriptive listings to a concise 'less is more' bulleted format sharply reduced user edit rates and increased sales conversion.
- [19:55](https://www.youtube.com/watch?v=Ej-Vnu5RxjA&t=1195s) Removing 'AI' branding from the feature name improved adoption and engagement by overcoming user AI fatigue and presenting the tool as practical utility.
- [24:05](https://www.youtube.com/watch?v=Ej-Vnu5RxjA&t=1445s) Commercial foundation model APIs remain more commercially viable than self-hosting due to rapid external price drops, speed improvements, and low maintenance overhead.
## Detailed walkthrough
### [00:04](https://www.youtube.com/watch?v=Ej-Vnu5RxjA&t=4s) The Friction of the C2C Listing Experience and Mercari's Scale
Max Frenzel introduces his background as the leader of Mercari's newly established Applied AI Research team, having previously directed all AI initiatives across both product and engineering until July. He centers his talk on a foundational challenge faced by consumer-to-consumer (C2C) marketplaces: the immense hurdle of listing an item. Operating essentially as a digital flea market, Mercari requires users wishing to sell an everyday object—such as a wristwatch—to confront a daunting blank page. An inexperienced seller must determine a catchy title, write an accurate description, select the right category, and research fair market pricing by digging through completed searches to see what sells and what languishes. While this friction deters casual sellers, listing volume remains the single most vital metric of health for any marketplace, directly governing Gross Merchandise Value (GMV).
To frame the operational scale of this challenge, Frenzel outlines Mercari’s market dominance in Japan compared to its growing presence in the United States. While still expanding its footprint in the US, Mercari is the clear market leader in Japan with approximately 23 million monthly active users (MAU)—meaning one in every four Japanese adults interacts with the platform at least once a month. Celebrating its 12th anniversary since launch, the marketplace facilitates $7.5 billion in annual GMV and has logged over 3 billion listings over its lifetime. Accounting for intra-day, weekly, and monthly seasonality, Mercari processes an average of 850 brand-new listings every single minute.
- **Monthly Active Users (Japan):** ~23 million (1 in 4 Japanese adults)
- **Annual Marketplace GMV:** $7.5 billion USD
- **Cumulative Platform Listings:** Over 3 billion items across 12 years
- **Listing Velocity:** ~850 new listings created per minute on average
---
[03:07](https://www.youtube.com/watch?v=Ej-Vnu5RxjA&t=187s) Activating the Long Tail: From "Never-Listers" to Confident Sellers
Mercari’s user base spans a wide continuum, ranging from semi-professional, high-volume merchants to complete amateurs and pure buyers who have never sold a single item. Because Mercari originated as an online flea market, its core mission has always centered on casual, light listers. Power sellers already have their workflows dialed in—sometimes even partially automated—and do not suffer from the intimidation of the blank page. In contrast, the vast long tail of hesitant users represents Mercari's highest-leverage growth opportunity. Converting a buyer or "never-lister" into an active seller generates a massive boost to their lifetime GMV.
Capturing these users requires securing "early wins." If a first-time seller has a frustrating, confusing, or sluggish experience, that negative impression is nearly impossible to undo, discouraging them from ever trying again. A successful onboarding journey requires speed, ease, and psychological reassurance—not only making the act of listing painless, but also ensuring the generated listing quickly converts into an actual sale.
---
[05:02](https://www.youtube.com/watch?v=Ej-Vnu5RxjA&t=302s) Rejecting the Chatbot: Great UX Doesn't Look Like AI
When exploring how generative AI could eliminate seller friction, the obvious question arose: should Mercari simply build an in-app conversational chatbot? The team observed that many sellers were already consulting ChatGPT externally to brainstorm listing details and craft item copy. However, the team deliberately rejected building a conversational interface. Chatbots can work in select niches, but turning a listing form into a multi-turn dialogue merely replaces one form of work with another, rather than removing the friction entirely. Mercari had previously tested a chat-based search interface for buyers and discovered how difficult it was to maintain meaningful user engagement.
Instead, Frenzel underscores a guiding product philosophy: "The best AI does not even look like AI to the user. It's really just great UX." Everyday consumers do not care whether an underlying system uses an LLM; they simply want a fast, effortless outcome. Forcing a chat wrapper onto every problem adds needless conversational overhead instead of streamlining the underlying task.
---
[06:45](https://www.youtube.com/watch?v=Ej-Vnu5RxjA&t=405s) Architecture of AI Listing: GPT-4o Mini and Intentional Friction
The resulting feature redesign transformed an arduous, multi-screen ordeal into a streamlined 3- to 4-tap flow: the seller snaps a photo, taps a confirmed category, lets the AI process for a brief moment, and receives an auto-populated description, title, and suggested price ready for immediate submission. When Mercari began developing the feature, they initially relied on OpenAI’s GPT-4o. The engineering team knew GPT-4o was far too slow and expensive to handle their production throughput of hundreds of listings per minute, but they chose to build anyway, betting that model costs and latencies would drop rapidly. That bet paid off: right before launch, OpenAI released GPT-4o mini. Mercari became an early adopter, achieving dramatic cost and latency reductions while simultaneously improving listing quality. Because GPT-4o had been so costly, the initial design had restricted the multimodal prompt to a single image; switching to GPT-4o mini made multi-image inputs economically viable, and the richer visual context elevated output quality far beyond what the larger model achieved on a single photo.
```mermaid
flowchart TD
accTitle: Mercari AI Listing Architecture Flow
subgraph Client ["Client Interface"]
A["User takes item photos"] --> B["Predict category candidates"]
B --> C["User taps & confirms category"]
end
subgraph AI_Engine ["Multimodal Processing"]
C --> D["Feed multiple images + confirmed category into GPT-4o mini"]
D --> E["Generate title, description, & pricing suggestions"]
end
subgraph Finalize ["Submission"]
E --> F["Review pre-filled listing"]
F --> G["Publish in 3-4 total taps"]
end
```
A crucial architectural choice was inserting an intermediate category prediction and confirmation step rather than passing images straight to the text generation prompt. This served two critical functions:
1. **Grounding and Disambiguation:** Grounding the LLM with a confirmed category tag prevents it from getting distracted by cluttered backgrounds (e.g., distinguishing a wristwatch on someone’s arm from the crowd of people or furniture behind it).
2. **User Agency and Trust:** While an extra tap might seem like added friction, testing proved that asking users to confirm an accurate category suggestion gave them a tangible sense of control. Total automation left users uneasy; this lightweight step fostered trust in the downstream AI-generated draft.
---
[10:55](https://www.youtube.com/watch?v=Ej-Vnu5RxjA&t=655s) Dogfooding, Iterative Evals, and Red-Teaming Unintended Sales
To define what constitutes a high-quality listing, Mercari conducted an extensive internal dogfooding campaign involving approximately 100 employees across two weeks, producing thousands of test listings. The team engineered a dedicated internal testing tool equipped with targeted feedback fields, asking evaluators specific questions such as what details the model hallucinated or what key product traits it omitted. An auxiliary LLM parsed and synthesized this feedback corpus, enabling rapid, data-informed prompt iterations. Frenzel notes that in generative AI development, reaching 90% quality is deceptively quick, but grinding from 90% to 99% or 100% requires exhaustive, time-consuming evaluation datasets.
Dogfooding also served as an adversarial red-teaming lab for Mercari’s first customer-facing generative output. Team members immediately stress-tested the boundaries by pointing cameras at each other to see if the tool would attempt to list a human for sale. The system appropriately defaulted to accessories, attempting to sell eyeglasses or clothing instead. Frenzel tested the system on his own pet dog at home; because Mercari strictly bans the sale of live animals, the category tree contained no animal classification. The category classifier defaulted to the nearest valid catalog item—a stuffed animal—and the downstream LLM framed the description accordingly, ensuring platform safety policies remained intact.
---
[14:03](https://www.youtube.com/watch?v=Ej-Vnu5RxjA&t=843s) Balancing Marketplace Stakeholders and Humanizing the Tone
As prompt tuning progressed, the team encountered a nuanced question: "What's a good listing, and good for whom?" A marketplace balances three competing interests: the seller (who needs confidence in the draft), the buyer (who needs clarity to make a purchase), and the search ranking algorithm (which dictates discoverability). Pleasing the algorithm alone would have led the AI to emulate experienced power sellers who engage in keyword stuffing—a spammy behavior Mercari actively discourages. The team had to strike an intentional compromise across all three constituents.
When the team finally placed the optimized tool in front of external users in dedicated test environments, they received unexpected feedback: the listings sounded *too* professional. The AI was churning out polished, corporate marketing copy reminiscent of an Amazon brand storefront. On a peer-to-peer flea market, overly slick commercial prose made buyers suspicious, leading them to believe the seller was "dodgy" and hiding product defects, while sellers felt self-conscious publishing text that did not sound like them. Product managers, designers, and AI engineers collaborated closely to tune the prompts toward a conversational, authentic, and human tone tailored to Mercari's community culture.
Following full deployment, the feature delivered sweeping performance gains across Mercari’s core business metrics:
- **Listing Conversion Rate (LCR):** Measurable uplift in the rate of platform visitors successfully publishing a listing.
- **Average Listings Per User (ALPU):** Substantial increase in items listed per user.
- **Core Demographic Impact:** The strongest lifts occurred precisely within the primary target segment—never-listers and infrequent, light sellers.
[17:04](https://www.youtube.com/watch?v=Ej-Vnu5RxjA&t=1024s) Post-Launch Nervousness and the "Less is More" Prompt Shift
Launching a generative AI feature at the scale of Mercari comes with significant public scrutiny. As a brand with tens of millions of users, the team monitored social channels like Twitter closely on launch day, feeling understandably anxious about edge cases—fearing scenarios where users might try to list outrageous items like babies or prompt the system into outputting offensive statements. Fortunately, thorough internal dogfooding and edge-case testing paid off: user feedback was overwhelmingly positive, and the launch went off without any major incidents.
Once the initial launch proved stable, the team re-evaluated their approach. In the first production iteration, Mercari optimized strictly for completeness. They intentionally erred on the side of generating long, well-rounded listings filled with descriptive text, fearing that omitted details would degrade listing quality. However, analyzing real user interaction logs revealed a critical flaw in this assumption: having to prune unwanted text or rewrite fluffy AI-generated descriptions created immense user friction. Modifying text retrospectively was far more frustrating than simply adding a missing specification.
```mermaid
flowchart TD
accTitle: Evolution of Prompting Strategy
subgraph V1["Initial Approach: Optimizing for Completeness"]
A1["Long, Verbose AI Text"] --> B1["User forced to delete fluff & rewrite"]
B1 --> C1["High friction & high edit rate"]
end
subgraph V2["Iterated Approach: Less is More"]
A2["Concise, Bullet-like Format"] --> B2["Only high-confidence facts included"]
B2 --> C2["Zero-edit 1-click listings & higher sales"]
end
```
Acting on this insight, the engineering team overhauled the prompt to be significantly less verbose. They transitioned to a clean, bulleted output structure and instructed the model to include only details it was exceptionally confident in, prioritizing omission over speculative filler. When evaluated in an A/B test, this single prompt redesign became the most impactful change post-launch:
- **Seller funnel improvements**: Listing Completion Rate (LCR) and Average Listings Per User (ALPU) both trended upward.
- **Marketplace efficiency**: Seller Conversion Rate (SCR—the proportion of listings that successfully sell) and Average Sales Per User (ASPU) rose noticeably.
- **Reduced edit behavior**: Counterintuitively, despite generating less total text, the edit rate plummeted; sellers routinely reviewed the AI output and hit publish without touching a single character.
---
[19:55](https://www.youtube.com/watch?v=Ej-Vnu5RxjA&t=1195s) Dropping the AI Label and Beating AI Fatigue
Following the success of the simplified prompt, the team ran an experiment around feature branding. The capability had originally been introduced to users under the Japanese moniker *AI出品* ("AI Shuppin", or AI Listing Support). Mercari launched an A/B test that stripped all AI terminology from the UI, renaming it to plain functional descriptors such as automatic listing or automatic title and description generation.
Removing the "AI" label triggered an immediate bump in user adoption and feature performance. This shift pointed to widespread "AI fatigue." With countless consumer products plastering AI onto every surface, users have grown skeptical of overhyped, underdelivering tech novelties. Presenting the workflow as an ordinary, dependable product utility helped regain user trust. By treating generative AI as invisible infrastructure rather than a flashy marketing gimmick, the feature felt less intimidating and far more practical.
---
[21:26](https://www.youtube.com/watch?v=Ej-Vnu5RxjA&t=1286s) Core Takeaways: Tomorrow’s AI, Prompt Evals, and Superpowers
Reflecting on the entire development lifecycle, three overarching principles emerge for building customer-facing generative AI:
1. **Design for tomorrow's AI capabilities**: Technology is advancing faster than standard product deployment cycles. Designing around the limits of current models risks building an obsolete product by launch. Teams should anticipate imminent model upgrades—just as Mercari architected for GPT-4o and seamlessly migrated to GPT-4o mini—and build for the capabilities that will exist three months down the road.
2. **Relentlessly test and iterate on prompts**: Crafting prompts is often the hardest and most underestimated engineering hurdle in production GenAI. Creating exhaustive, high-quality evaluation datasets serves as both an operational compass and a competitive moat, enabling teams to benchmark and iterate on prompts with velocity.
3. **Hide the AI, reveal the magic**: End users ultimately care about frictionless problem resolution, not the underlying architecture. Teams should resist the temptation to spotlight AI for vanity and instead let the speed and smoothness of the user experience speak for itself.
Product leaders should locate the friction points in their customer journey and resist defaulting to simple chatbots or passive automation. Instead of just replacing human action, the more potent approach is giving users "superpowers"—providing intuitive tools that instill the confidence and trust needed to solve complex tasks independently. Mercari documents these architectural paradigms and organizational structures across detailed engineering blog posts for teams navigating similar paths.
---
[24:05](https://www.youtube.com/watch?v=Ej-Vnu5RxjA&t=1445s) Q&A: Custom In-House Models vs. Frontier APIs at Scale
During the audience Q&A, an attendee asked whether Mercari plans to transition away from commercial APIs toward self-hosted or custom-trained models to mitigate latency and operational costs.
The response highlighted the delicate trade-offs between self-hosting and managed frontier APIs:
- **The hidden cost of self-hosting**: While self-hosting smaller open-source models can appear cheaper on raw unit economics, it carries massive engineering maintenance overhead. Self-hosted models remain frozen in time unless continuously fine-tuned, maintained, and served on expensive infrastructure.
- **The managed API advantage**: Commercial foundation model APIs (such as OpenAI) offer continuous "free upgrades" in quality, speed, and periodic price cuts with minimal migration overhead beyond light prompt adjustments.
- **Valuable data for targeted fine-tuning**: With roughly half of Mercari’s entire inventory flow running through this system—translating to approximately 400 listings generated every minute—the team captures a rich stream of user edits and successful submissions. This data represents prime material for domain-specific fine-tuning.
- **The current verdict**: Mercari continues to benchmark API-based fine-tuning against self-hosted variants, but current pricing and model velocity make self-hosting flagship multimodal listing models difficult to justify commercially.
## Notable quotes
> [05:02](https://www.youtube.com/watch?v=Ej-Vnu5RxjA&t=302s) The best AI does not even look like AI to the user. It's really just great UX. — Max Frenzel
> [14:03](https://www.youtube.com/watch?v=Ej-Vnu5RxjA&t=843s) What's a good listing, and good for whom? — Max Frenzel
> [17:04](https://www.youtube.com/watch?v=Ej-Vnu5RxjA&t=1024s) Less is More — Max Frenzel
> [19:55](https://www.youtube.com/watch?v=Ej-Vnu5RxjA&t=1195s) AI出品 — Max Frenzel
> [19:55](https://www.youtube.com/watch?v=Ej-Vnu5RxjA&t=1195s) AI fatigue — Max Frenzel
> [21:26](https://www.youtube.com/watch?v=Ej-Vnu5RxjA&t=1286s) Design for tomorrow's AI capabilities — Max Frenzel
> [21:26](https://www.youtube.com/watch?v=Ej-Vnu5RxjA&t=1286s) Relentlessly test and iterate on prompts — Max Frenzel
> [21:26](https://www.youtube.com/watch?v=Ej-Vnu5RxjA&t=1286s) Hide the AI, reveal the magic — Max Frenzel
> [21:26](https://www.youtube.com/watch?v=Ej-Vnu5RxjA&t=1286s) superpowers — Max Frenzel
> [24:05](https://www.youtube.com/watch?v=Ej-Vnu5RxjA&t=1445s) free upgrades — Max Frenzel
## Video description
Max Frenzel, Head of Applied AI Research, Mercari
Abstracts
How can generative AI move beyond chatbots to solve core customer experience problems? At Mercari, we recognized that the manual effort of listing items was a significant barrier for our users, and one of the key friction points limiting the number of items on our C2C marketplace. This talk will detail the development of our AI Listing feature, which uses multimodal AI to turn photos into complete listings in seconds, dramatically increasing listing volume and improving the seller experience for our 23M+ monthly active users. I will cover our approach to system design, prompt engineering, UX considerations, user feedback integration, and the impressive results we've seen. Join me to gain actionable strategies for implementing AI to enhance your own customer journeys and achieve measurable business impact.
Mercari eliminated core seller friction by replacing intimidating listing forms with an invisible, multimodal AI workflow that prioritizes user agency over conversational interfaces.
Marketplace Friction at Scale (0:04)
Consumer-to-consumer (C2C) platforms depend fundamentally on listing volume to drive Gross Merchandise Value (GMV), but casual users often abandon sales when confronted with blank forms requiring titles, descriptions, categories, and price research. For Mercari—which reports approximately 23 million monthly active users in Japan, $7.5 billion in annual GMV, and an average flow of 850 new listings per minute—converting hesitant "never-listers" into active sellers represents its most significant growth lever.
Power sellers often possess established, semi-automated listing habits, whereas first-time sellers require immediate, reassuring wins. A negative initial experience risks permanently discouraging casual users, making speed and psychological ease the primary requirements for onboarding.
Rejecting Chatbots for Invisible UX (5:02)
Although users often turned externally to tools like ChatGPT to draft listing copy, Mercari intentionally rejected building an in-app conversational chatbot. Product leadership determined that multi-turn dialogue creates unnecessary friction by merely swapping one task for another, a finding reinforced by past experiments with chat-based buyer search that struggled with retention.
The resulting flow relies on the principle that effective AI should resemble seamless product design rather than an overt technology demo. Users simply snap photos, confirm a predicted category, and receive a pre-populated draft with a suggested price, title, and description ready for publication within three to four taps.
Architecture and Intentional Control (6:45)
Mercari originally prototyped the system using OpenAI's GPT-4o, anticipating that future model costs and latencies would decrease. The launch of GPT-4o mini validated this approach, enabling Mercari to affordably process multiple image inputs rather than a single photo, which substantially improved generation quality.
The architecture deliberately inserts a category prediction confirmation step before sending multimodal inputs to the model. This extra tap provides two distinct benefits: it grounds the language model against visual clutter—such as distinguishing a wristwatch from a busy background—and provides users with a sense of control that fosters trust in the subsequent automated text.
Tone Calibration and Conciseness (14:03)
During dogfooding and user testing, Mercari discovered that early AI drafts produced overly polished, corporate prose resembling commercial storefront copy. In a peer-to-peer flea market, slick marketing language alienated buyers by raising suspicions of hidden defects, while sellers felt the text lacked authenticity. The team subsequently tuned prompts to reflect a casual, human tone.
Post-launch analytics revealed that users found editing overly detailed descriptions more tedious than filling in missing information. Shifting to a concise, bulleted prompt that included only high-confidence details caused edit rates to plummet while boosting listing completion rates, average listings per user, and seller conversion rates.
Overcoming AI Fatigue (19:55)
Mercari initially marketed the feature in Japan as "AI Shuppin" (AI Listing Support). In subsequent A/B testing, removing "AI" terminology and relabeling the feature with straightforward functional terms—such as automatic listing—produced an immediate lift in user adoption.
This shift indicated clear "AI fatigue" among consumers wary of overhyped technology. Positioning the system as dependable infrastructure rather than an AI novelty increased user trust, reinforcing the strategy of using advanced models silently to give users practical capabilities without product friction.
Mercari routes roughly half of its total catalog volume—approximately 400 listings per minute—through this system, yielding proprietary interaction data suitable for fine-tuning. However, leadership notes that the rapid performance upgrades and price reductions of commercial foundation model APIs currently make the high maintenance and operational overhead of self-hosting multimodal models difficult to justify commercially.
## Background
本视频由科技会议频道 Ai4 于 2025 年 8 月 27 日发布在 YouTube 平台，内容为 Mercari 应用 AI 研究主管 Max Frenzel 的主题演讲。根据页面简介，演讲深入剖析了 Mercari 如何跳出传统 Chatbot 框架，利用多模态生成式 AI 打造一键式商品上架功能，从而化解 2300 万月活用户面临的繁琐刊登痛点，实现显著的业务增长与体验跃升。
## TL;DR
针对二手交易平台中普通用户面临的“面对空白页上架难”痛点，Mercari 开发了基于多模态大模型的 AI Listing 功能，用户只需拍照并轻触确认类目，即可自动生成标题、极简描述与建议价格。团队在技术上敏捷切入 GPT-4o mini 架构，通过类目锚定与严苛的内部 Dogfooding 评测消除机器感与合规风险。迭代过程中，团队反直觉地发现“精简且高置信度的列表式描述”以及“完全剥离 AI 营销标签”反而能大幅降低用户编辑率并赢得信任，成功激活了长尾卖家的上架与成交意愿。
## Atmosphere & style
整场演讲呈现出高度务实、严谨且极具工业实战底色的技术分享氛围。主讲人以沉稳清晰的叙事节奏娓娓道来，既有大厂面对数千万级用户规模时的审慎与敬畏（如上线前针对极端案例的红队测试），又充满对极致用户体验打磨的敏锐洞察。从技术架构演进到“隐藏 AI 显露魔法”的哲学反思，内容干货密集且层层递进，为听众提供了极富启发性的前沿产品落地全景。
## Key takeaways
- [00:04](https://www.youtube.com/watch?v=Ej-Vnu5RxjA&t=4s) Mercari 日本站拥有约 2300 万月活用户且平均每分钟生成 850 个新商品，但面对空白页撰写标题、描述和定价是普通用户上架商品的最大阻力。
- [03:07](https://www.youtube.com/watch?v=Ej-Vnu5RxjA&t=187s) AI 功能的核心目标客群是长尾的“轻度与非卖家”，通过打造极简的首次上架体验提供“早期胜利”，能显著提升该群体在全生命周期内的 GMV 贡献。
- [05:02](https://www.youtube.com/watch?v=Ej-Vnu5RxjA&t=302s) 明确拒绝 Chatbot 对话式交互：最好的 AI 体验不应把操作负担转化为打字聊天负担，而应在幕后彻底消除繁琐操作，让用户感受不到 AI 的刻意存在。
- [06:45](https://www.youtube.com/watch?v=Ej-Vnu5RxjA&t=405s) 采取“面向未来技术构建”策略，从早期的 GPT-4o 顺利过渡到 GPT-4o mini，利用小型模型大幅降低延迟与成本，并通过多图并发输入提升生成质量。
- [08:56](https://www.youtube.com/watch?v=Ej-Vnu5RxjA&t=536s) 在流程中设计“系统预测类目 + 用户单次点击确认”机制，不仅为多模态模型提供了抗噪锚点，还赋予了用户关键的掌控感与信任度。
- [10:55](https://www.youtube.com/watch?v=Ej-Vnu5RxjA&t=655s) 大模型产品从 90% 到 99% 的突破极其艰难，Mercari 通过大规模员工 Dogfooding 和红队测试构建专属评估集，利用大模型归纳反馈闭环迭代 Prompt。
- [14:03](https://www.youtube.com/watch?v=Ej-Vnu5RxjA&t=843s) 文案去除“过度专业广告腔”：C2C 场景下过于华丽的公关用语会引发买家防备心，调优为自然真诚的个人口吻反而大幅提升了购买转化。
- [17:33](https://www.youtube.com/watch?v=Ej-Vnu5RxjA&t=1053s) Prompt 策略从“追求完整”大刀阔斧重构为“极简列表与宁缺毋滥”，减少用户删减废话的心理负担，不仅降低了人工修改率，还驱动人均销售额同步上涨。
- [19:55](https://www.youtube.com/watch?v=Ej-Vnu5RxjA&t=1195s) 去概念化的信任红利：在界面中剔除所有“AI”营销标签、回归纯粹功能描述，能有效消除用户的“AI 疲劳”并提升功能采用率。
- [21:27](https://www.youtube.com/watch?v=Ej-Vnu5RxjA&t=1287s) 演讲者提出 GenAI 落地三原则：面向明天的 AI 能力设计、严苛测试与迭代 Prompt（构建评估集护城河）、隐藏 AI 并聚焦赋予用户解决问题的超级力量。
- [24:05](https://www.youtube.com/watch?v=Ej-Vnu5RxjA&t=1445s) 自托管模型维护成本高昂且处于能力冻结状态，当前采用前沿商业 API 能持续享受降本增效红利；AI 承接 50% 刊登所沉淀的用户修改闭环数据是未来模型微调的核心资产。
## Detailed walkthrough
### [00:04](https://www.youtube.com/watch?v=Ej-Vnu5RxjA&t=4s) 卖家上架难题与 Mercari 的市场规模
在 C2C 二手交易平台（如 Mercari 在线跳蚤市场）中，上架商品（listing）对普通用户而言是一道巨大的门槛。卖家面对的是一个空白页面，需要构思标题、组织描述、设定合理价格，甚至还要去搜索同类商品看看别人怎么写、哪些卖得好。然而，上架商品数量是衡量二手交易平台健康状况及平台成交总额（GMV）最核心的指标之一。
Mercari 在日本拥有庞大的市场渗透率，月活跃用户（MAU）达到约 2300 万，相当于每 4 位日本成年人中就有 1 位每月至少使用一次。平台年 GMV 达到 75 亿美元，成立 12 年以来累计上架商品超过 30 亿件。在考虑日内、每周与每月的周期波动后，平台平均每分钟生成约 850 个新上架商品。如何利用 AI 大幅简化上架流程、激发用户发布更多商品，成为了核心探索课题。
- **日本月活用户（MAU）**：约 2,300 万（渗透率达 1/4 日本成年人）
- **年平台成交总额（GMV）**：75 亿美元
- **累计上架商品量**：超 30 亿件（成立 12 周年）
- **上架生成速率**：全平台平均约 850 件/分钟
---
### [03:07](https://www.youtube.com/watch?v=Ej-Vnu5RxjA&t=187s) 目标客群定位：激活长尾“轻度与非卖家”
平台的卖家群体呈现出两极分化：一端是经验丰富、甚至具备半专业性质的重度卖家（Heavy sellers），另一端则是长尾的轻度卖家（Light listers，可能使用多年但仅发布过一两件商品）以及从未发布过商品的纯买家（Never listers）。重度卖家往往拥有固定的上架流程甚至自动化工具，不存在“面对空白页”的困扰；而 Mercari 本质上是大众在线跳蚤市场，轻度卖家构成的长尾客群具备极高价值。
将一个“从未上架过的用户”转化为“上架卖家”，能够大幅提升该用户在生命周期内的总 GMV 贡献。对于新卖家而言，“首次体验”至关重要：如果第一次上架受挫，将很难挽回并促使其再次尝试；而流畅的上架体验和快速出单的“早期胜利”（early wins），能建立用户对平台的信心。
---
### [05:02](https://www.youtube.com/watch?v=Ej-Vnu5RxjA&t=302s) 拒绝 Chatbot：最好的 AI 体验甚至不像 AI
面对这一痛点，团队首先思考是否应当直接做一个类似于 ChatGPT 的对话式助手（Chatbot），因为不少用户本就会主动使用 ChatGPT 来辅助写商品文案。但团队很快否定了这一方案：聊天界面并非在所有场景下都意味着便捷；把一个原本的操作问题转化为打字聊天的问题，往往不是最优解。
团队此前曾探索过面向买家的对话式搜索，发现如果没有极其精细的设计，很难有效维持用户参与度。真正的体验优化应当是最大程度地“消除繁琐操作”，而不是给用户换一种交互形式去承担工作。核心理念在于：“最好的 AI 甚至让用户感觉不到 AI 的存在，它本质上就是出色的用户体验（UX）。”用户不在乎功能是否贴了 AI 标签，他们只在乎体验是否足够惊艳。
```mermaid
flowchart TD
accTitle: AI 上架流程架构图
accDescription: 展示用户从上传图片到确认类目，再由多模态 LLM 生成最终商品详情的流程。
subgraph UserInput["用户端操作"]
A["拍摄/上传商品多张图片"] --> B["系统预测候选类目"]
B --> C["用户轻触确认类目\n(赋予掌控感与信任)"]
end
subgraph LLMProcessing["多模态大模型处理"]
C --> D["输入：多图 + 确认的类目"]
D --> E["GPT-4o mini 多模态推理"]
end
subgraph Output["最终输出"]
E --> F["自动填充标题、描述与建议价格"]
F --> G["3-4 次轻触即可提交发布"]
end
```
---
### [06:45](https://www.youtube.com/watch?v=Ej-Vnu5RxjA&t=405s) 技术演进与架构设计：从 GPT-4o 到 GPT-4o mini
最终落地的 AI Listing 功能将原本需要多次点击以及繁重打字描述的流程，压缩至仅需 3 到 4 次轻触操作。用户只需拍摄商品图片、选择系统推荐的类目，AI 稍作计算便会自动生成标题、详细描述和建议价格，用户确认即可一键发布。
在开发初期，团队使用的是 OpenAI 的 GPT-4o 模型。尽管当时团队清楚 GPT-4o 的调用成本和延迟过高、无法在生产环境大规模落地，但他们选择“面向未来的技术进行构建”，相信大模型迭代的速度。在功能即将上线前，OpenAI 推出了 GPT-4o mini，Mercari 成为首批在大规模业务中接入该模型的团队之一。
切换到小型模型不仅降低了延迟与成本，更通过系统架构的调整反向提升了输出质量：
- 在 GPT-4o 阶段，受制于成本和速度，系统只能基于用户上传的第 1 张图片进行单图分析；
- 换用 GPT-4o mini 后，系统可以支持多图并发输入。尽管单模型体量较小，但多张图片提供的丰富视角与细节信息，大幅提升了最终生成的商品文案质量。
---
### [08:56](https://www.youtube.com/watch?v=Ej-Vnu5RxjA&t=536s) 类目预测与人工介入的微妙平衡
在整个流程中，系统设计了一个关键中间环节：先基于图像预测类目树中的分类，让用户轻触确认类目后，再将“图片 + 确认的类目”共同输入 LLM 生成文案。这一看似增加了摩擦的交互设计带来了两大核心收益：
1. **精准定位与抗噪（Grounding）**：明确的类目给多模态大模型提供了锚点。在复杂或杂乱的背景中（例如在一堆杂物或人群前拍摄手表），模型能准确锁定主体物品，避免对无关背景进行错误描述。
2. **心理掌控感与信任构建**：由于类目预测准确度极高，用户只需单手轻点一次。这不但没有引起用户反感，反而赋予了用户对生成过程的“控制感”（Agency），显著增强了对 AI 生成结果的信任度。
---
### [10:55](https://www.youtube.com/watch?v=Ej-Vnu5RxjA&t=655s) 内部 Dogfooding 评测与红队测试
在提示词工程（Prompt Engineering）优化阶段，团队组织了约 100 名内部员工进行了为期两周的高强度内部试用（Dogfooding），生成了数千条测试数据。团队专门开发了一个带有反馈机制的内部工具，收集员工关于“文案中多写了什么（模型幻觉/多余添加）”与“漏写了什么（未识别特征）”等具体反馈，并利用另一个 LLM 自动归纳分析这些反馈，辅助迭代提示词。
大模型落地开发中，“从 0 到 90% 往往非常容易和迅速，但从 90% 提升到 99% 乃至 100% 才是真正艰难且耗时的过程”，而严密的评估集（Evaluation Datasets）是突破这最后瓶颈的关键。
此外，试用期间还进行了红队测试（Red Teaming）。例如员工尝试对着同事拍照并尝试“出售同事”，系统通常会根据类目预测机制自动聚焦在其佩戴的眼镜或配饰上。在演讲者测试对着自家宠物狗拍照时，由于 Mercari 平台严禁销售活体动物且类目树中没有“动物”分类，类目预测模块自动退守到最接近的分类（如“毛绒玩具/Stuffed Animal”），从而避免了违规活体商品上架的合规风险。
---
### [14:03](https://www.youtube.com/watch?v=Ej-Vnu5RxjA&t=843s) 平衡多方利益与消除“机器感”
在打磨商品文案质量时，团队面临着“好文案是对谁而言”的三方权衡难题：
- **卖家端**：文案需要让卖家感到准确并有信心发出；
- **买家端**：文案需要具备足够吸引力并促成购买转化；
- **搜索引擎端**：需要契合站内搜索算法的权重与排序规则。
如果单纯迎合搜索引擎，AI 可能会学会部分老卖家堆砌关键词（Keyword stuffing）的不良习惯。因此必须在三者之间取得适度平衡。
当内部测试完成并邀请真实用户进行小规模内测时，团队收到了一条意料之外的反馈：文案听起来“过于专业”。在 C2C 平台中，交易双方是普通人对普通人，此前由 AI 生成的华丽文案更像亚马逊（Amazon）上的专业品牌公关广告词。买家看到过于光鲜的文案反而会起疑心，怀疑二手个人卖家在隐瞒商品缺陷或进行虚假宣传。最终，产品经理、设计师和 AI 工程师协同对 Prompt 进行了大幅调优，让语气回归自然、真诚的个人口吻。
该功能正式上线后取得了显著成效，平台核心指标迎来全面提升：
- **上架转化率（LCR, List Conversion Rate）**：显著提升；
- **人均上架数（ALPU, Average Listings Per User）**：显著增加；
- 在**从不上架用户与轻度卖家**（核心目标客群）中表现尤为亮眼，成功激活了长尾供给。
### [17:04](https://www.youtube.com/watch?v=Ej-Vnu5RxjA&t=1024s) 上线首日的紧张氛围与用户反馈
在产品正式上线当天，团队内部其实承受着巨大的心理压力。作为一家体量庞大的知名平台，Mercari 必须时刻紧盯社交媒体与 Twitter（现 X）上的实时动态。团队甚至开玩笑且不无担忧地猜测：会不会有人拿这个功能去给活生生的婴儿上架，或者引导 AI 生成某些极其可怕、违规的内容？所幸，得益于上线前充分的内部试用（Dogfooding）和对各种极端边缘情况（Edge Cases）的严格测试，实际发布非常平稳，没有发生任何重大事故，用户反馈也极为积极。
首战告捷后，团队迅速回到工作台前，探讨如何在此基础上做进一步的体验提升与优化。
### [17:33](https://www.youtube.com/watch?v=Ej-Vnu5RxjA&t=1053s) 从“追求完整”到“极简精准”的 Prompt 重构
在最初推出的第一个版本中，团队的设计目标是“追求完整度（Optimizing for completeness）”。当时的做法宁可让 AI 多写一些，哪怕带有一点无意义的套话和废话（Fluff），也希望生成一篇面面俱到、看起来丰满详尽的商品描述。
然而，在深入分析用户行为日志与数据后，团队提出了一个全新的假设：当 AI 生成了过多的冗余内容时，用户被迫逐字阅读、删减或重写，这种“事后修改”的心智负担和痛苦感，远高于“内容略有缺失、只需用户手动随手补上一两句”。
```mermaid
flowchart TD
accTitle: Prompt 优化逻辑对比
subgraph V1["第一代策略：追求完整度 (Completeness)"]
A1["生成详尽但冗长的描述"] --> B1["包含套话与不确定细节"]
B1 --> C1["用户被迫通读并删除多余废话"]
C1 --> D1["操作负担大、修改率高"]
end
subgraph V2["第二代策略：极简与高置信度 (Conciseness)"]
A2["生成简洁的列表式描述"] --> B2["仅保留极高置信度事实"]
B2 --> C2["一目了然，无需大量删改"]
C2 --> D2["零修改直发、转化率与销售双提升"]
end
```
基于这一假设，团队对 Prompt 进行了大刀阔斧的重构：大幅降低啰嗦程度，采用更简单清晰的列表（Bullet points）格式，并严格设定准则——**宁缺毋滥**。AI 只能包含自己具有极高置信度、且确信是用户所需的信息；凡是不确定的细节一律省略，绝不强行编造。
这一改动进行了 A/B 测试，结果成为了上线以来效果最显著的一次单一优化：
- **Listing 转化指标全线提升**：不仅刊登转化率（LCR, Listing Conversion Rate）和人均刊登量（ALPU, Average Listings Per User）保持增长，后端的卖家成交流水指标——卖家转化率（SCR, Seller Conversion Rate）以及人均销售额（ASPU, Average Sales Per User）也同步上升。这意味着商品不仅更容易挂牌，而且卖得更快、更好。
- **编辑率反常识下降**：团队原先以为简化的描述会促使更多用户手动添加细节，但实际数据显示整体编辑率明显降低。大量卖家拍照后，直接看着 AI 生成的极简列表点击发布，实现了“拍照即上架”的零摩擦体验。
### [19:55](https://www.youtube.com/watch?v=Ej-Vnu5RxjA&t=1195s) 隐藏 AI 标签：去概念化带来的信任红利
除了精简 Prompt，另一个看似微小却带来巨大正向反馈的实验是**去除所有“AI”标签**。在功能上线初期，该特性在日语界面中被宣传为“AI 出品サポート”（AI 刊登支持）。
随后团队做了一组 A/B 测试，将产品界面中所有带有“AI”的字眼完全剥离，改用朴素的功能性描述（例如“自动刊登”或“自动生成标题与描述”）。测试结果显示，仅仅去掉了 AI 标签，功能的使用率和转化表现就立即获得了提升。
这揭示了当前消费级产品中普遍存在的“AI 疲劳（AI Fatigue）”现象。市场上充斥着各种言过其实、强贴 AI 标签的产品，反而容易透支用户信任或让用户产生不切实际的过高期望。当把炫技的 AI 外壳剥除、还原为纯粹顺畅的工具体验时，用户反而更加安心和信任。“Hide the AI, reveal the magic（隐藏 AI，显露魔法）”成为了团队沉淀出的重要产品哲学。
### [21:27](https://www.youtube.com/watch?v=Ej-Vnu5RxjA&t=1287s) 核心复盘与给开发者的建议
在总结这一路的产品探索时，演讲者提炼了三条面向 Generative AI 应用开发者的核心建议：
1. **面向明天的 AI 能力进行设计（Design for tomorrow's AI capabilities）**：AI 技术迭代极其迅速。如果完全受限于当下的模型成本和能力，等到产品经过数月开发准备上线时，技术基座可能已经落后。要有预判未来技术趋势的勇气（如在 GPT-4o 昂贵时就预判并等待小模型 GPT-4o mini 的降临），提前为即将到来的能力构建架构。
2. **严苛测试并持续迭代 Prompt（Relentlessly test and iterate on your prompts）**：Prompt 工程和评估是生成式 AI 最难、也最容易被低估的环节。搭建扎实、高质量的评估数据集（Eval Datasets）不仅是快速调优 Prompt 的前提，更是抵御外部竞争的坚实商业护城河（Moat）。
3. **隐藏 AI，展现魔法（Hide the AI, reveal the magic）**：用户并不关心背后用了什么模型或算力，他们只在意自身体验是否丝滑。产品应当让用户为体验买单，而不是为概念买单。
在结尾处，演讲者向全场观众抛出了一个反思问题：在你们自己的客户旅程中，最大的阻力（Friction）究竟在哪里？与其机械地思考“如何用 Chatbot 自动化”，不如思考“如何赋予用户超级力量（Superpowers）”，让用户在技术的支持下以极低的门槛和极高的自信，亲手解决原本繁琐的问题。
### [24:05](https://www.youtube.com/watch?v=Ej-Vnu5RxjA&t=1445s) Q&A 环节：自建模型与商业 API 的权衡
在随后的问答环节中，有观众提问 Mercari 是否考虑出于成本和延迟的考量，放弃商业 API 转而部署自研或自托管的专用模型（Custom Models）。
演讲者给出了非常务实的解答：
- **自托管模型的隐藏成本巨大**：虽然自建或微调开源模型在初期推理成本上看似更便宜，但后续的维护成本极其高昂——不仅消耗算力与金钱，更会牵扯团队大量宝贵的时间。此外，自建模型的技术状态往往是“冻结”的，除非持续投入重新训练，否则无法自动享受到前沿基座模型的进步。
- **商业 API 的免费升级红利**：使用 OpenAI 等前沿 API 平台，虽然需要随着版本更迭微调 Prompt，但平台自身会持续提供“性能自动升级”与“价格大幅下调”的红利。
- **高质量数据的微调前景**：目前 Mercari 平台大约 50% 的刊登都是通过该 AI 功能完成的，这意味着系统每分钟都在产生约 400 条包含“原始图像、AI 生成草稿、用户人工修改对比、最终刊登与成交结果”的高价值全链路数据。这些数据是未来针对特定刊登任务进行模型微调的极佳养料。团队目前正在探索基于 API 的微调以及特定场景下的私有化方案，但在现阶段，全面自托管推理依然很难在投入产出比上成立。
## Notable quotes
> [05:02](https://www.youtube.com/watch?v=Ej-Vnu5RxjA&t=302s) 最好的 AI 甚至让用户感觉不到 AI 的存在，它本质上就是出色的用户体验（UX）。 — Max Frenzel
> [10:55](https://www.youtube.com/watch?v=Ej-Vnu5RxjA&t=655s) 从 0 到 90% 往往非常容易和迅速，但从 90% 提升到 99% 乃至 100% 才是真正艰难且耗时的过程。 — Max Frenzel
> [17:33](https://www.youtube.com/watch?v=Ej-Vnu5RxjA&t=1053s) 宁缺毋滥。AI 只能包含自己具有极高置信度、且确信是用户所需的信息；凡是不确定的细节一律省略，绝不强行编造。 — Max Frenzel
> [19:55](https://www.youtube.com/watch?v=Ej-Vnu5RxjA&t=1195s) Hide the AI, reveal the magic（隐藏 AI，显露魔法）。 — Max Frenzel
> [21:27](https://www.youtube.com/watch?v=Ej-Vnu5RxjA&t=1287s) 面向明天的 AI 能力进行设计（Design for tomorrow's AI capabilities）。 — Max Frenzel
> [21:27](https://www.youtube.com/watch?v=Ej-Vnu5RxjA&t=1287s) 严苛测试并持续迭代 Prompt（Relentlessly test and iterate on your prompts）。 — Max Frenzel
> [21:27](https://www.youtube.com/watch?v=Ej-Vnu5RxjA&t=1287s) 搭建扎实、高质量的评估数据集（Eval Datasets）不仅是快速调优 Prompt 的前提，更是抵御外部竞争的坚实商业护城河（Moat）。 — Max Frenzel
> [21:27](https://www.youtube.com/watch?v=Ej-Vnu5RxjA&t=1287s) 与其机械地思考‘如何用 Chatbot 自动化’，不如思考‘如何赋予用户超级力量（Superpowers）’，让用户在技术的支持下以极低的门槛和极高的自信，亲手解决原本繁琐的问题。 — Max Frenzel
## Mentioned
Max Frenzel, Mercari, OpenAI, GPT-4o, GPT-4o mini, ChatGPT, Amazon, Twitter (X), Ai4
## Video description
Max Frenzel, Head of Applied AI Research, Mercari
Abstracts
How can generative AI move beyond chatbots to solve core customer experience problems? At Mercari, we recognized that the manual effort of listing items was a significant barrier for our users, and one of the key friction points limiting the number of items on our C2C marketplace. This talk will detail the development of our AI Listing feature, which uses multimodal AI to turn photos into complete listings in seconds, dramatically increasing listing volume and improving the seller experience for our 23M+ monthly active users. I will cover our approach to system design, prompt engineering, UX considerations, user feedback integration, and the impressive results we've seen. Join me to gain actionable strategies for implementing AI to enhance your own customer journeys and achieve measurable business impact.
[Source](https://www.youtube.com/watch?v=Ej-Vnu5RxjA)
Source quotes
[5:02] Max Frenzel: 最好的 AI 甚至让用户感觉不到 AI 的存在，它本质上就是出色的用户体验（UX）。
[10:55] Max Frenzel: 从 0 到 90% 往往非常容易和迅速，但从 90% 提升到 99% 乃至 100% 才是真正艰难且耗时的过程。
[17:33] Max Frenzel: 宁缺毋滥。AI 只能包含自己具有极高置信度、且确信是用户所需的信息；凡是不确定的细节一律省略，绝不强行编造。
[19:55] Max Frenzel: Hide the AI, reveal the magic（隐藏 AI，显露魔法）。
[21:27] Max Frenzel: 面向明天的 AI 能力进行设计（Design for tomorrow's AI capabilities）。
[21:27] Max Frenzel: 严苛测试并持续迭代 Prompt（Relentlessly test and iterate on your prompts）。
[21:27] Max Frenzel: 搭建扎实、高质量的评估数据集（Eval Datasets）不仅是快速调优 Prompt 的前提，更是抵御外部竞争的坚实商业护城河（Moat）。
[21:27] Max Frenzel: 与其机械地思考‘如何用 Chatbot 自动化’，不如思考‘如何赋予用户超级力量（Superpowers）’，让用户在技术的支持下以极低的门槛和极高的自信，亲手解决原本繁琐的问题。
Source quotes
[5:02] Max Frenzel: The best AI does not even look like AI to the user. It's really just great UX.
[14:03] Max Frenzel: What's a good listing, and good for whom?
[17:04] Max Frenzel: Less is More
[19:55] Max Frenzel: AI出品
[19:55] Max Frenzel: AI fatigue
[21:26] Max Frenzel: Design for tomorrow's AI capabilities
[21:26] Max Frenzel: Relentlessly test and iterate on prompts
[21:26] Max Frenzel: Hide the AI, reveal the magic
[21:26] Max Frenzel: superpowers
[24:05] Max Frenzel: free upgrades
Original transcript
[0:04] Hi everyone, thanks for making it to the
[0:06] session. I'm Max. I currently lead the
[0:08] applied AI research team that we just
[0:10] newly established. Before that, I was
[0:12] leading all of AI at Mercer, both on the
[0:15] product and on the engineering side
[0:16] until July this year. And today, I want
[0:19] to share with you some of the learnings
[0:21] from one of the projects that my teams
[0:23] worked on during this time, especially
[0:25] around the listing experience for our
[0:28] users.
[0:31] Next slide please.
[0:36] So the listing experience is one of the
[0:39] most critical events on any kind of
[0:42] marketplace especially a CTOC
[0:45] marketplace. So basically an online flea
[0:47] market that's essentially what we are
[0:50] listing is really big hurdle for our
[0:52] users. If I want to say let's for
[0:55] example sell this watch right I need to
[0:57] go on the page I basically start from
[0:59] this blank page I need to figure out
[1:01] what do I actually want to write what's
[1:03] a good title what's a good description
[1:05] what's the right price I probably going
[1:07] to go into search look for similar
[1:08] listings see what other people wrote um
[1:11] see what sells well what doesn't sell
[1:14] well especially for our less experienced
[1:17] users that's a huge huge hurdle but the
[1:20] number of listings is one of the single
[1:23] most valuable indicators for the health
[1:26] of any kind of marketplace and
[1:28] ultimately for the GMD that flows
[1:30] through the marketplace. So what we
[1:32] asked ourselves is what if this process
[1:35] could be easier? Can we use AI to
[1:38] simplify this process for our users?
[1:41] To give you a little bit of context
[1:42] because here in the US we do have a
[1:44] subsidiary here and we do have a local
[1:46] marketplace in the US but I don't think
[1:49] we're well established brand here yet.
[1:51] However, in Japan, we are the market
[1:54] leader. We have around 23 million
[1:56] monthly active users. That means 1 in
[1:59] four Japanese adults uses us at least
[2:02] once a month. So, we have a very strong
[2:04] penetration in Japan. Our annual GMV, so
[2:07] the money that flows through the
[2:08] marketplace last year was 7.5 billion
[2:11] US. And to date, and I think we just
[2:14] celebrated the 12th anniversary since
[2:16] the app's original launch, we had more
[2:19] than 3 billion listings on our
[2:21] marketplace in total. And this number
[2:24] keeps accelerating, of course, as we're
[2:26] onboarding more user, as we're growing.
[2:28] Um, this number of listings keeps
[2:30] increasing at an accelerating rate.
[2:33] Right now accounting for all the
[2:35] seasonality both within the day but also
[2:37] across the week and months we have on
[2:39] average around 850 new listings being
[2:42] generated per minute. So again think
[2:45] about the difficulty of this listing
[2:47] process starting from that blank page.
[2:50] What if we can make this massively
[2:52] simpler? Can we encourage our existing
[2:55] users to create more and better
[2:57] listings? And can we also bring users in
[3:00] who were maybe just buyers and never
[3:02] created a listing before to activate
[3:04] them into the listing process. And
[3:07] that's really a key category for us. So,
[3:10] as you can imagine, we have a huge
[3:12] distribution of the kinds of users that
[3:13] we have on our platform. From the very
[3:15] heavy sellers, almost kind of semi-p
[3:18] profofessional sellers to the complete
[3:20] amateurs, light listers who maybe have
[3:22] been users for years but maybe listed
[3:25] one or two items to people who are just
[3:27] buyers on the marketplace and never
[3:28] listed before. Originally, we were built
[3:32] as kind of online free market. So, our
[3:34] target users were exactly these light
[3:36] listers. Also, even though we have this
[3:40] significant fraction of heavy users, the
[3:43] long tail of light listers is extremely
[3:46] valuable to us. So, we were wondering,
[3:49] can we especially make the experience
[3:51] easier for these people? Because the
[3:53] heavier users, they don't really have
[3:54] that blank page problem, right? They
[3:56] have their process down. Maybe they even
[3:58] have it automated to some extent. But
[4:00] the people who are inexperienced who
[4:01] maybe never listed are only very
[4:03] occasionally these are the ones that we
[4:05] really want to encourage to list more by
[4:08] making the process easier and giving
[4:10] them more confidence in the process.
[4:13] This single activation is one of the
[4:15] highest value activations for us.
[4:17] Converting a user from a neverlister to
[4:19] a listister because it massively
[4:21] increases the lifetime GMBB from a
[4:23] particular user. And early rins are
[4:27] extremely important. If a user has a bad
[4:30] first experience, it's very very hard to
[4:33] undo that impression and give them a
[4:35] better feeling about it and encourage
[4:37] them to retry. So giving them a
[4:39] successful first experience both in
[4:41] terms of the listing experience itself
[4:43] but then also that listing converting
[4:45] into a sale is extremely important for
[4:48] us and the ease, speed and confidence
[4:50] that we can give to our users in this
[4:53] process is extremely important. So we
[4:56] had identified this problem and asked
[4:59] ourselves can we use AI to improve that.
[5:02] Now, the first thing that you might ask
[5:04] yourself when you figured out there's a
[5:06] problem and I want to AI use AI to solve
[5:08] it is should I just build a chatbot? And
[5:10] we knew a lot of our users were actually
[5:13] actively using chat GPT as kind of
[5:15] assistant when they wanted to list
[5:16] something they had a conversation with
[5:18] chat GPT to figure out their listing.
[5:21] So, we're wondering should we just bring
[5:22] that experience into our app itself by
[5:26] building our own chat experience? But we
[5:29] very quickly settled on no. And I hope
[5:32] the fact that you're here at this
[5:33] conference probably means also that many
[5:35] of you are already kind of beyond this
[5:36] initial stage. But chat bots can work,
[5:39] but they're rarely the best solution.
[5:42] Especially if you think you want to
[5:43] create an better, more easy experience
[5:46] for the user. Just shifting one problem
[5:49] into a chat problem is actually not
[5:51] necessarily the right thing to do. And
[5:54] chat doesn't necessarily mean ease.
[5:55] We've also separately explored a
[5:58] chatbased search experience for our
[5:59] buyers and we found it very very
[6:02] difficult to engage users properly if
[6:04] you don't design this well. So chats
[6:06] often should not be the first thing you
[6:09] want to try. Ideally you want to remove
[6:12] work as much as possible, not reframe it
[6:14] into a different problem. And the key
[6:17] thing here is and I'm going to come back
[6:19] to this several times. The best AI does
[6:22] not even look like AI to the user. It's
[6:24] really just great UX.
[6:27] This build
[6:29] ultimately your users don't really care
[6:32] um if a feature is AI. They just want to
[6:34] have an amazing experience, right? So
[6:38] focus on the experience and don't feel
[6:40] like you need to turn everything into a
[6:43] chat experience.
[6:45] Okay. So once we thought this through,
[6:47] here's what the feature actually looks
[6:48] like in practice. This video is very
[6:50] quick, but I'll show you the slides in a
[6:52] moment. But essentially user takes an
[6:55] image of what they want to sell. They
[6:57] select the right category. The AI thinks
[6:59] for a bit and automatically description
[7:02] and price are set and you can literally
[7:04] just submit your listing. So here's that
[7:07] flow again. What would have taken
[7:09] previously
[7:10] many many taps, not even including
[7:13] writing the title and the description,
[7:15] we brought down to three or four tap
[7:17] interactions until you can actually
[7:20] submit your listing. So what we did in
[7:25] the process as we were developing this
[7:27] and this is a really interesting lesson
[7:28] as well. We started developing this for
[7:33] or using OpenAI's um GPT40 model. We
[7:37] knew at the time for sure that 40 would
[7:40] be too expensive and too slow for us to
[7:43] launch this at the volume that we need
[7:45] to launch it for our users. We still
[7:47] went ahead and developed the feature
[7:50] based on this trusting in the technology
[7:52] to evolve along the way. Just before we
[7:54] were ready to launch the feature, OpenAI
[7:57] launched for O mini and we were one of
[7:59] the first to switch to that and deploy
[8:01] it at scale. And that not only allowed
[8:04] us to bring down the cost and the
[8:07] latency to a level that makes sense from
[8:09] like a business perspective, but it
[8:11] actually also improved the quality. And
[8:14] the reason why is that initially we just
[8:17] allowed users to submit um a single
[8:20] image as the input or they could submit
[8:22] multiple images but we only based the AI
[8:24] suggestion on the first image because
[8:26] everything else submitting multiple
[8:27] images would have been completely
[8:29] unreasonable from a cost and latency
[8:31] perspective with 40. However, with the
[8:34] smaller model we could actually
[8:35] immediately submit multiple images into
[8:38] the flow. And even though the model
[8:39] itself was smaller and weaker, the
[8:42] additional images massively boosted the
[8:44] quality. So I think that's a key lesson,
[8:47] built for the technology that's just
[8:48] around the corner, not today's
[8:50] technology and expect it to catch up
[8:52] with you because it's evolving so fast
[8:54] right now. Also, one thing I think I
[8:56] didn't fully explain and we'll get to
[8:58] more details in a moment. The flow is
[9:01] this. So images get submitted by the
[9:03] user. Then we actually have an
[9:05] intermediary step where we predict the
[9:07] categories of the image um from within
[9:09] our category tree and we have the user
[9:12] confirm this category and then both the
[9:16] images and the category get fed into the
[9:19] LLM as context for generating the actual
[9:22] final listing. And this category
[9:26] prediction and confirmation step is
[9:28] super important. And there's two reasons
[9:30] why. The first one is maybe the obvious
[9:33] one. Adding the correct category and
[9:35] feeding that into the LLM as additional
[9:38] context and grounding massively improves
[9:41] the quality of the output listings.
[9:43] Right? If there's an image like let's
[9:45] say maybe a watch is easy, but like
[9:47] here's a watch and I label this as
[9:49] watch. The model knows exactly what to
[9:52] look for. Especially if it's like a
[9:53] cluttered background. Let's say I take
[9:54] an image of this with you guys in the
[9:56] background and other items around. If I
[9:58] know it's the category watch, the model
[10:00] knows exactly what to focus on and what
[10:02] to actually create a listing around
[10:03] versus all the other possibilities that
[10:05] may be cluttering the image. So this
[10:08] accuracy boost, that's maybe the obvious
[10:10] one. What's less obvious, this
[10:12] additional small step of user
[10:15] interaction, it might seem like friction
[10:17] because it's an additional thing for the
[10:19] user to do, but it actually in our tests
[10:21] massively improve the user experience
[10:23] because it's simple enough and our
[10:26] category predictions are usually right.
[10:28] So, the user just has to tap on one of
[10:30] the suggested categories, but it gives
[10:33] the user a feeling of being in control,
[10:36] and that's incredibly valuable. And it
[10:38] also increases the trust in the actual
[10:41] AI suggestions in the end. So not all
[10:44] the steps should be completely automated
[10:46] and removed. This really gives the user
[10:49] a sense of being having some agency and
[10:51] being in control of the process.
[10:55] As we were developing this feature more
[10:57] and more, one of the key questions that
[10:59] we kept asking ourselves is what
[11:01] actually is a good listing on our
[11:04] marketplace? And as we were sort of
[11:06] refining the prompt that was creating
[11:08] those listings, we did extensive dog
[11:11] fooding internally. We had around 100
[11:13] employees over two weeks test this
[11:16] feature and I don't remember how many
[11:18] thousands of test listings we created.
[11:20] We built a special internal tool just
[11:23] for testing this feature, just for this
[11:25] dog fooding process that had the listing
[11:28] flow. But then once the listing was
[11:31] created, there was also a custom
[11:32] feedback form where we asked people for
[11:35] example
[11:36] um what would you remove from this
[11:38] listing or what should would you add to
[11:40] this listing so we knew what the LLM was
[11:43] maybe missing from the pictures or what
[11:45] it kind of made up or added um that
[11:48] maybe wasn't desired by the user and
[11:50] there were a few other fields like this
[11:52] and then we used another LLM to actually
[11:54] analyze all this feedback and help us
[11:56] iteratively improve on the prompts of
[11:59] the model. So establishing these
[12:01] evaluations um and yeah that's another
[12:03] thing we did actually in this process
[12:05] creating our evaluation data sets for
[12:06] later prompt tweaking super valuable and
[12:10] it's painful it's very time consuming
[12:13] but it's maybe the single most important
[12:14] thing you can do because with this kind
[12:17] of prompt tweaking LLM based development
[12:20] getting to 90% super easy super quick
[12:24] getting from 90% to 99 or 100 that's
[12:26] where the real pain and difficulty lies
[12:28] but that's where the eval valuations are
[12:30] super critical. Another thing that this
[12:32] dog fooding really helped us with was
[12:35] red teaming. Um this was our first big
[12:38] feature where we had direct LLM
[12:41] generated output shown to the user and
[12:44] exposed on the platform. So of course
[12:46] there's always this worry about what are
[12:48] the potential risks and one of the first
[12:51] things all of us did when we had the
[12:53] feature in our hands and tried it out
[12:54] was pointing it at each other and
[12:55] basically saying hey what happens if I
[12:57] try and sell you? Luckily, you usually
[12:59] focus on some accessories. So, if he
[13:01] pointed at me, probably would try and
[13:02] sell my glasses. And again, actually
[13:04] having this category prediction step in
[13:07] between massively helped with that. One
[13:10] thing I tried at home was try and point
[13:12] it at my dog. Can I sell my dog or can I
[13:14] have this model sell my dog? Of course,
[13:16] selling animals is not legal on our
[13:19] marketplace. Uh so, it shouldn't create
[13:21] a listing for a dog. Luckily, the
[13:24] category prediction step because there
[13:26] is no animal category in our category
[13:29] tree, it defaulted to what it thought
[13:31] was the next most likely thing. Um, so
[13:33] it actually for the most likely was I
[13:35] think stuffed animal, which fair enough.
[13:39] Um, there were a few other predictions,
[13:40] but I had to choose something that is in
[13:42] our category tree. And then in the
[13:44] generation step, right, the LLM gets fed
[13:47] the image plus the category that it is.
[13:49] So, if I had selected stuffed animal, it
[13:51] would just describe this as if it was a
[13:53] stuffed animal, not something you're not
[13:55] supposed to sell on the platform. Um,
[13:58] and a lot of these things we figured out
[13:59] and tested during the dog fooding
[14:01] process as well.
[14:03] And then the last maybe a bit more
[14:05] subtle question that we addressed during
[14:07] all of this or that we faced which we
[14:10] weren't actually thinking too much at
[14:11] least not to that extent going into this
[14:13] whole process was when we ask ourselves
[14:16] what's a good listing we need to ask
[14:18] good for whom because there's different
[14:20] people and entities involved in this
[14:22] right is it good for the seller meaning
[14:25] it gives them confidence that they want
[14:27] to list this thing that was just
[14:29] generated is it good for the buyer so if
[14:32] someone finds this item, do they likely
[14:34] convert into um a sale or is it good for
[14:38] our search algorithm? So, does it rank
[14:40] high in the search ratings? If it's not
[14:42] good for that, no buyer will ever see
[14:44] it, even if it would be the perfect item
[14:46] for the buyer. In an ideal world, all
[14:48] three of these are the same. But the
[14:50] reality is that you actually have to
[14:51] compromise a little bit between those.
[14:53] For example, we know some of our more
[14:55] experienced users are gaming our search
[14:57] algorithms through keyword stuffing and
[14:59] similar things. If we had just optimized
[15:01] the AI for the search algorithm, we
[15:03] would have basically trained it to have
[15:05] this bad behavior that we want to get
[15:06] rid of on the platform. So you have to
[15:08] make some trade-offs and balances around
[15:11] these things.
[15:13] Once we did all this internal testing
[15:14] and internal dog fooding, uh we were
[15:17] ready to put it in front of real users.
[15:18] And again, we invited a few users in. We
[15:22] had like a special test environment set
[15:23] up for this. And we got really, really
[15:26] valuable feedback. At this point, we
[15:28] were super confident in our listings. We
[15:30] felt they sounded amazing. They sounded
[15:32] so professional, so correct. What our
[15:35] users told us, actually, they sound too
[15:37] professional. We're CTOC marketplace.
[15:39] It's real humans selling to real humans.
[15:41] Essentially, our AI sounded more like
[15:43] Amazon listings, right? It sounded like
[15:46] professional sales copy of like a brand
[15:48] trying to advertise their item. That's
[15:50] not what our users want. The sellers
[15:52] didn't feel comfortable submitting this
[15:54] and buyers didn't feel comfortable
[15:56] because they thought uh this is a used
[15:58] item from a real user. The way they
[16:00] write it, this does not match. This
[16:02] seems dodgy. They're hiding something.
[16:04] So, we had to basically teach the model
[16:06] through prompt tuning to have a more
[16:09] human natural um tone. And that was a
[16:14] big collaborative effort between our
[16:15] product managers, our designers, and the
[16:17] AI engineers all working together to get
[16:20] the right tone. um and really tweak this
[16:23] user experience to what people are
[16:25] expecting from the Meccani marketplace.
[16:29] So all of these tests we were confident
[16:31] to launch and we did launch and it was a
[16:34] really great success for us. So we saw
[16:36] key a lift in all the key metrics
[16:38] especially what we were targeting was
[16:40] LCR so list conversion rate how many of
[16:42] our users convert into listers and ALPU
[16:46] average listings per user. Both of these
[16:49] went up um significantly and especially
[16:52] with our key target user group the never
[16:55] and light listers that's where it was
[16:57] particularly successful and we saw this
[16:59] significant increase in the average
[17:01] number of listings from these user
[17:04] groups also as a brand as big as us uh
[17:07] we always keep an eye on social media
[17:09] and Twitter and we were quite nervous
[17:12] that day if people are I don't know
[17:14] listing their babies or something or
[17:16] trying to get it say something
[17:17] absolutely absolutely horrible and
[17:19] luckily overall there was very positive
[17:21] user feedback and no major incidents. So
[17:23] also our dog fooding and testing of all
[17:26] the kind of edge cases uh proved solid
[17:30] and it was a good success. Once we had
[17:33] the successful launch we kind of went
[17:35] back to the drawing board and we asked
[17:37] ourselves how can we further improve
[17:40] this feature. The first iteration that
[17:43] we created and that we launched, we
[17:44] tried to aim for completeness and
[17:47] optimize for completeness. Meaning we
[17:49] were ering on the side of having the AI
[17:51] generate maybe a bit too much, maybe a
[17:54] bit too much fluff as well, but have it
[17:55] like a nice well-rounded listing uh
[17:58] rather than missing anything. But as we
[18:01] were looking at the data and the logs,
[18:03] we kind of came up with this hypothesis
[18:05] that editing a listing retrospectively,
[18:08] so let's say the AI generated it, but
[18:10] there's too much in there. Having to go
[18:12] in and remove things or rewrite things
[18:14] is a much more painful experience than
[18:17] if it was missing something and you just
[18:18] have to add a point. So with that
[18:21] hypothesis, we tried a revised prompt um
[18:25] that basically was much less verbose. Um
[18:30] it had much simpler structure in the
[18:31] output almost like a bullet list and we
[18:34] were airing on the side of only include
[18:37] things that you're very very confident
[18:39] are actually correct and that the user
[18:40] wants rather omit something than add
[18:42] something you're not completely sure
[18:44] about. And when we launched this as an
[18:46] AB test again it was extremely
[18:48] successful it was the single biggest
[18:50] change that we made um after the initial
[18:53] launch uh in terms of the effect. So
[18:55] again, the original LCR, ALPU, both were
[18:58] positive, but also the SCR, the seller
[19:00] conversion rates, how many listers
[19:02] actually end up selling their item. And
[19:04] the ASPU, the average sales per user,
[19:07] went up. So not only was it positive for
[19:09] the listers in the listing flow, making
[19:12] it more confident to list, the item
[19:13] actually also sold better. And what was
[19:16] interesting is overall the edit rate
[19:18] went down. You might expect because we
[19:21] included less information in the listing
[19:23] and that was our assumption. Users would
[19:25] then go in and add more of their own
[19:27] information which might have been the
[19:29] case to some extent but overall the edit
[19:31] rate went down. So this means more users
[19:33] went straight from taking a pictures
[19:35] having the AI generate the listing and
[19:38] then just hitting the list button
[19:39] straight away without any edits and
[19:41] still the sales rates actually went up.
[19:44] So sometimes less is more and that's
[19:46] definitely what we found in this case.
[19:48] Don't over complicate it. Make it
[19:50] simpler if you can and then improve from
[19:53] there.
[19:55] One more test that we launched which was
[19:59] very simple in a way but again had a
[20:01] very big impact was dropping the AI
[20:03] label. Initially we advertised this as
[20:07] AI shippto. So basically AI listing
[20:09] support in Japanese.
[20:12] After a while, we tried an AB test,
[20:15] dropping any AI labeling on the feature.
[20:18] So, I don't remember what exactly we're
[20:20] calling it now during this test, but
[20:23] basically was something like automatic
[20:24] listing or automatic title description
[20:27] generation. So, completely removing any
[20:29] reference to AI. And what was
[20:32] interesting was immediately just with
[20:34] this different wording, the feature
[20:36] performed better. More users were
[20:38] adopting it. And I think this is a
[20:41] really key lesson as well. There is so
[20:43] much AI fatigue right now, especially
[20:46] among users. Everyone's labeling
[20:48] everything as AI this, AI that. Users
[20:52] lose a lot of trust. They don't really
[20:54] know often what it means because also
[20:56] often there's like overpromise. So
[20:58] dropping this AI label I think gave
[21:01] users more confidence to actually use
[21:03] the feature. It didn't distract us like
[21:05] something shiny new that they didn't
[21:06] want. Um, it was again just a good user
[21:09] experience. So that's also something to
[21:11] keep in mind. Maybe you don't
[21:14] necessarily need to advertise that
[21:15] you're using AI even if you yourself
[21:18] proud of your usage of AI. Your users
[21:20] might not necessarily care and without
[21:22] the label they might even trust you
[21:24] more. And that kind of brings me to the
[21:27] end and the key lessons that I want to
[21:29] share with you from this uh some key
[21:31] takeaways. So the first one is design
[21:34] for tomorrow's AI capabilities. That's
[21:36] kind of our lesson from the original
[21:38] development where we work with 40 but
[21:40] then switch to 40 mini as it launched.
[21:43] The field is evolving so quickly right
[21:45] now. If you starting with today's
[21:47] capabilities and developing around
[21:49] those, by the time you're launching your
[21:50] feature in 3 months or whatever it might
[21:53] be, it might already be outdated. So if
[21:56] you can have the courage and if you
[21:57] think you understand what's just across
[21:59] the horizon, you can actually develop
[22:01] for that and expect it to come and be
[22:04] ready by the time you're ready to
[22:06] launch. So have the courage to think
[22:08] just a little bit ahead.
[22:10] The next one is relentlessly test and
[22:12] iterate on your prompts. That's probably
[22:15] when you're working with generative AI
[22:17] the single most difficult and
[22:19] underestimated
[22:21] um challenge that you're going to be
[22:22] facing. Your evaluations are one of the
[22:25] most important things that you're going
[22:26] to be working on and they're also really
[22:29] like a solid mode actually that protects
[22:32] you a little bit from the competition if
[22:33] you create really good evaluation data
[22:36] sets that then allow you to quickly
[22:38] iterate on your prompts. And the last
[22:41] one is hide the AI, reveal the magic.
[22:44] Your users care about the user
[22:46] experience. They don't care if it's AI.
[22:48] Even if you're super proud of your AI,
[22:51] just deliver a good experience and yeah,
[22:55] have the users celebrate the experience,
[22:57] not the AI. With that, I want to leave
[23:00] you with a question. Um, where does the
[23:02] friction live in your customer
[23:04] experience? I'm sure all of you might
[23:06] have things that come to mind straight
[23:08] away. And the first question that you
[23:10] then might ask yourself is, okay, how
[23:12] can I automate this process? And maybe
[23:14] also ask yourself, can I use a chatbot?
[23:15] And maybe the answer should be no. But
[23:17] another question rather than just
[23:19] thinking about how can I automate this
[23:21] is also yeah rather than automating can
[23:24] I somehow enable my users to give them
[23:27] superpowers to deal with that problem by
[23:29] themselves maybe the problem doesn't
[23:30] need to go away but can I enable my user
[23:33] to solve this problem with much more
[23:35] ease giving them the confidence giving
[23:37] them the trust in your technology to
[23:39] overcome this problem by themselves so
[23:43] I'm excited to see what you guys will
[23:44] all be building for your users um If
[23:47] you're curious to hear more about the
[23:48] feature itself as well as the
[23:50] development process, uh we have a bunch
[23:52] of articles on our blog about this
[23:54] feature, about other features, also
[23:56] about how we structure the org around
[23:57] developing these things. Um so yeah,
[24:00] feel free to check that out and thank
[24:02] you.
[24:05] >> From a cost and latency perspective, um
[24:08] are you thinking about deploying any
[24:10] custom models or are you sticking to
[24:13] >> Really, really good question. something
[24:14] that we've thought about for a very long
[24:17] time. We have for other LLM based
[24:20] features deployed custom models. Um this
[24:23] one is one of more flagship kind of
[24:25] models where we expect to need more
[24:27] heavy models. And the thing is once you
[24:30] deploy a model it might be cheaper in
[24:32] the short term but then you have the
[24:33] maintenance cost which is a huge pain
[24:35] not just like time uh money but also the
[24:37] time that you need to invest in it. Also
[24:39] the technology is basically frozen in
[24:40] place right unless you keep retraining
[24:42] that model. you stuck with it at one
[24:44] point. Whereas if you're using something
[24:47] like OpenAI API, your model
[24:50] automatically upgrades by itself in a
[24:52] way. Sure, you need to do some prompt
[24:53] tweaking as the models iterate through
[24:55] their generations, but you kind of
[24:57] almost get these free upgrades or price
[24:59] drops at the same time. So for this kind
[25:02] of thing, we're not doing that. However,
[25:04] we're now collecting so much data,
[25:06] right? We're having right now, I think
[25:08] around 400, so half of our listings
[25:10] roughly are generated through this
[25:11] features. So 400 listings per minute of
[25:15] user generating a listing, potentially
[25:18] editing that listing, and then
[25:20] submitting it. This is incredibly
[25:22] valuable data for fine-tuning a model on
[25:25] this particular task. And there we're
[25:28] still exploring different options from
[25:30] API based fine-tuning to our own models,
[25:33] but it's a very difficult balance to
[25:36] justify self-hosting at the moment.
