Pfizer Deploys Agentic AI to Replace Marketing Analytics Queues

COCO: An Agentic AI Assistant for Marketing Analytics With Pfizer — Ai4

Ai4

Generative text-to-SQL tools carry unacceptable error rates for enterprise decision-makers, pushing Pfizer to combine deterministic metric extraction with knowledge graphs to diagnose commercial bottlenecks in seconds.

AI-assisted summary of COCO: An Agentic AI Assistant for Marketing Analytics With Pfizer — original source. Check the original for full context.

Source published:

Detailed notes

## Background Published by the Ai4 channel on YouTube on August 19, 2026, this presentation features Shanshan Bi, Director of Commercial AI Acceleration at Pfizer. The session details COCO, an agentic AI assistant developed internally to resolve enterprise marketing analytics bottlenecks by translating natural language queries into automated anomaly detection, multi-source evidence gathering, and human-readable insights.

## TL;DR To eliminate the multi-week ad hoc SQL request cycles that delay strategic adjustments for commercial brands like Paxlovid, Pfizer built COCO, an agentic analytics engine. The system executes a structured four-stage pipeline: scanning KPIs to isolate outliers, generating plausible root-cause hypotheses via an enterprise knowledge graph, cross-validating empirical evidence across claims and detailing feeds, and delivering actionable interventions with estimated ROI. Due to governance constraints, token bloat in fragile 'Uber prompts', and an unacceptable 10% error margin in generative text-to-SQL chatbots, the architecture leverages deterministic backend metric extraction while keeping human marketers in the loop and augmenting existing data science workflows.

## Atmosphere & style The presentation blends conversational stand-up wit with a pragmatic, transparent engineering narrative. Bi engages the audience by contrasting the high-level promise of generative AI with the messy reality of enterprise data silos, prompt maintenance, and colleague anxieties about job security. The session feels like an authentic, behind-the-scenes debrief that grounds cutting-edge agentic workflows in real-world business constraints.

## Key takeaways - [00:04](https://www.youtube.com/watch?v=Twct4OtyuZ4&t=4s) Enterprise commercial dashboards track thousands of KPIs but only provide retrospective visibility, failing to explain root causes or prescribe operational next steps. - [02:35](https://www.youtube.com/watch?v=Twct4OtyuZ4&t=155s) The traditional ad hoc analytics loop between marketers and data scientists requires weeks of turnaround for each query, causing a severe speed-to-insights bottleneck. - [04:42](https://www.youtube.com/watch?v=Twct4OtyuZ4&t=282s) Pfizer's COCO engine streamlines analytics via a four-stage pipeline: automated anomaly detection, knowledge graph hypothesis generation, cross-dataset evidence gathering, and action recommendations. - [04:42](https://www.youtube.com/watch?v=Twct4OtyuZ4&t=282s) COCO is intentionally architected with a human-in-the-loop model, requiring human marketers to evaluate and sign off on suggested interventions rather than automating decisions. - [06:56](https://www.youtube.com/watch?v=Twct4OtyuZ4&t=416s) By synthesizing multi-source data across claims, CDC feeds, and detailing transcripts, the system successfully identified HCP dosing education gaps as the root driver for depressed Paxlovid uptake in elderly DDI cohorts. - [08:32](https://www.youtube.com/watch?v=Twct4OtyuZ4&t=512s) Insights are delivered through lightweight HTML reports anchored on patient conversion funnels, using clear visual markers for deviations, empirical proof, and exportable execution plans. - [10:28](https://www.youtube.com/watch?v=Twct4OtyuZ4&t=628s) Unconstrained conversational text-to-SQL chatbots face severe hurdles in regulated pharma environments due to PII compliance risks, governance rules, and prompt fragility. - [12:31](https://www.youtube.com/watch?v=Twct4OtyuZ4&t=751s) A 90% accuracy rate in autonomous SQL generation leaves a 10% margin of error that undermines marketer trust, prompting a shift toward deterministic backend KPI extraction. - [13:30](https://www.youtube.com/watch?v=Twct4OtyuZ4&t=810s) Using massive 'Uber prompts' packed with schema mappings and few-shot templates drives up token costs while still struggling to eliminate the final 1–5% SQL hallucination rate. - [15:06](https://www.youtube.com/watch?v=Twct4OtyuZ4&t=906s) Positioning agentic AI as an accelerating assistant for data science teams rather than a replacement helps alleviate internal workforce concerns. - [16:00](https://www.youtube.com/watch?v=Twct4OtyuZ4&t=960s) The outlier detection mechanism utilizes a two-year historical baseline to run continuous time-series tracking, year-over-year comparisons, and peer-to-peer cohort benchmarking.

## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=Twct4OtyuZ4&t=4s) The Commercial Analytics Bottleneck: Drowning in Dashboards

Shanshan Bi introduces herself with a lighthearted disclaimer: although she moonlights doing stand-up comedy, she will not be cracking jokes about Pfizer today because "they will fire me." Shifting to serious business, she outlines the presentation around COCO, an agentic AI assistant built internally at Pfizer to resolve commercial marketing analytics bottlenecks. Her talk covers three main pillars: current operational pain points, COCO's architectural design and benefits, and an interactive demonstration.

To frame the problem, she asks the audience how brand marketers gauge whether a drug's commercial performance is succeeding and whether marketing strategies are effective. The conventional answer is tracking Key Performance Indicators (KPIs) visualized in dashboards. However, at a pharmaceutical enterprise like Pfizer, a single commercial brand such as Paxlovid (Pfizer’s COVID-19 antiviral therapy) involves thousands of discrete KPIs spread across dozens of distinct dashboards. Maintaining this ecosystem requires dedicated external vendor teams handling ad hoc requests alongside internal data science teams tasked with extracting insights.

In practice, marketers lack the hours in a day to comb through thousands of individual metrics. Even when they scan high-level dashboard summaries, dashboards provide visibility only into what has already taken place—they fail to explain the underlying mechanics or direct next steps. When commercial executives notice declining treatment adoption rates in a specific geographic region and ask why it occurred and how the team should respond, standard business intelligence tools fall silent. Answering those two critical questions—*why* something happened and *what to do next*—is the core design goal behind Pfizer's agentic AI.

[02:35](https://www.youtube.com/watch?v=Twct4OtyuZ4&t=155s) The Painful Query Loop and the "Speed to Insights" Problem

To illustrate the current friction, Bi walks through a standard Paxlovid marketing query. If a marketer wants to know the overall percentage of eligible patients treated with Paxlovid over the prior quarter, the query is straightforward: they open the dashboard and locate the number within thirty seconds. The breakdown occurs as soon as they pose natural follow-up questions, such as drilling into specific regional sub-cohorts—like untreated patient groups in the Northeast—to uncover why those patients never initiated therapy.

Because standard dashboards lack pre-aggregated tiles for every nuanced cohort, the marketer must open a ticket with an internal data scientist or external vendor. The data science team typically requests at least a week to coordinate with data engineering pipelines, author SQL queries, pull data extracts, interpret signals, assemble presentation decks, and schedule a formal read-out meeting. When the marketer finally reviews the deck, the findings prompt immediate follow-up inquiries regarding root causes and key drivers. The analyst then requests another week to schedule a follow-up review.

This iterative loop repeats continuously across commercial operations. By the time an initial question receives an actionable explanation, the quarter has already elapsed, leaving brand managers unable to adjust commercial strategy in flight. Bi stresses that this failure is fundamentally not a data deficiency—the data points and KPI definitions already exist. Rather, the enterprise suffers from a severe "speed to insights problem."

```mermaid flowchart TD accTitle: The Traditional Commercial Analytics Bottleneck subgraph S1["1. Standard Dashboard"] A["Marketer checks basic KPI<br/>(30 seconds)"] --> B["Identifies cohort anomaly<br/>(e.g., Northeast untreated)"] end

subgraph S2["2. Ad Hoc Request Queue"] B --> C["Requests deep dive from Data Scientist / Vendor"] C --> D["Data Engineering & SQL pipelines<br/>(1-week delay)"] D --> E["Slide deck created & review meeting held"] end

subgraph S3["3. Delayed Decision-Making"] E --> F["Follow-up question on root drivers<br/>(Another 1-week delay)"] F --> G["Quarter ends before strategy can adjust"] end ```

[04:42](https://www.youtube.com/watch?v=Twct4OtyuZ4&t=282s) COCO's Four-Step Architecture: From Anomaly to Action

To eliminate this weeks-long feedback loop, Pfizer developed COCO: a knowledge graph-powered insights engine that autonomously scans metrics, diagnoses anomalies, and generates concrete, ready-to-act recommendations. Rather than behaving like an open-ended conversational bot, COCO operates as an automated engine executing a structured four-stage workflow:

```mermaid flowchart LR accTitle: COCO Four-Step Processing Engine subgraph Pipeline["COCO Engine Core Pipeline"] direction TB N1["1. Scan KPIs & Detect Outliers<br/>(Finds 14-20 anomalies)"] --> N2["2. Query Knowledge Graph<br/>(Retrieve candidate explanations)"] N2 --> N3["3. Gather Empirical Evidence<br/>(Cross-examine disparate datasets)"] N3 --> N4["4. Present Human-in-the-Loop Actions<br/>(Recommendations with ROI estimates)"] end ```

The system executes this process through four sequential steps:

1. **Automated Anomaly Detection:** Specialized agents review thousands of KPIs across historical trailing windows (such as the past two weeks or thirty days) to identify statistically significant outliers—typically isolating 14 to 20 actionable deviations that a human would miss. 2. **Knowledge Graph Hypothesis Generation:** Instead of guessing randomly, the system consults an enterprise knowledge graph to retrieve multiple plausible candidate hypotheses that could explain the identified anomaly. 3. **Automated Evidence Gathering:** Agents independently validate or disprove each candidate hypothesis by interrogating Pfizer's underlying data systems, hunting down empirical evidence across patient cohorts (e.g., evaluating female patients over age 50 enrolled in Medicaid). 4. **Action Recommendations with ROI:** The platform surfaces prioritized interventions for brand teams, complete with projected return on investment (ROI) metrics. Marketers can accept recommendations, discard options, record operational rationale, or launch targeted A/B tests.

Bi emphasizes that COCO does not operate autonomously without oversight. The tool is intentionally architected around human-in-the-loop decision-making: "We are not develop the COCO as the AI telling what to do next... and we still need human marketers to sign off. We are not directly replace anyone's job."

[06:56](https://www.youtube.com/watch?v=Twct4OtyuZ4&t=416s) Case Study: Unpacking Paxlovid and Drug-Drug Interactions

Bi grounds the technical workflow in a real-world Paxlovid analytical challenge: explaining unexpectedly depressed prescription initiation rates among patients over 65 who were flagged with Drug-Drug Interaction (DDI) contraindication warnings.

When COCO examines this outlier cohort, its knowledge graph proposes several distinct candidate explanations: - Systemic data quality degradation or incomplete electronic health record feeds. - Patient-level hesitancy driven by fears that Paxlovid might adversely react with their maintenance medications. - Healthcare provider (HCP) knowledge deficits regarding clinical dosage adjustment protocols specifically formulated for co-medicated cohorts.

```mermaid flowchart TD accTitle: Paxlovid DDI Investigation Flow A["Outlier Detected: Patients 65+ with DDI flags<br/>exhibit low treatment rates"] --> B{"Knowledge Graph Hypotheses"} B --> C["Data quality / pipeline errors"] B --> D["Patient fear of interaction"] B --> E["HCP lack of dosing adjustment knowledge"]

E --> F["Data Synthesis Across Enterprise Feeds"] subgraph Datasets["Cross-Dataset Evidence Synthesis"] direction LR G1["Claims & Patient Data"] G2["HCP Detailing Logs"] G3["CDC / Wastewater / Hospital Data"] G4["Omnichannel & Target Lists"] G5["ML Model Inferences"] end F --- Datasets Datasets --> H["Validated Finding: HCP dosing education gap"] H --> I["Action: Adjust HCP target list & field detailing"] ```

To determine which hypothesis holds true, COCO's agents systematically query and correlate massive multi-source datasets: - Longitudinal medical and pharmacy claims. - Patient-level demographic records. - Healthcare Provider (HCP) master files and commercial target lists. - Public epidemiological tracking feeds from the CDC, including regional hospitalization metrics and wastewater surveillance data. - Field representative face-to-face detailing transcripts and multichannel marketing logs. - Predictive machine learning model outputs.

By cross-referencing these feeds, the system rules out data pipeline corruptions and demonstrates that the primary bottleneck stems from HCP prescribing uncertainty. Having substantiated the root cause with empirical data, the platform presents the commercial team with targeted actions, such as dynamically updating field rep call plans or delivering educational omnichannel assets on dose adjustment.

[08:32](https://www.youtube.com/watch?v=Twct4OtyuZ4&t=512s) Inside the COCO Interface: Funnels, Evidence, and Feedback

Demonstrating the user-facing interface, Bi explains that COCO avoids the bloat of traditional dashboards and heavy JavaScript applications, delivering insights instead through a lightweight, structured HTML report. The visual interface organizes findings into an intuitive exploratory hierarchy:

- **The Patient Conversion Funnel:** The top of the report anchors on a macro conversion funnel tracking sequential patient volumes: individuals testing positive or getting infected, total treated cohorts, patients initiated on Paxlovid, and secondary abandonment points where patients fail to pick up prescriptions at the pharmacy. - **Visual Marker Conventions:** - Findings and anomalous deviations are represented along conversion funnel layers as **purple triangles**. Clicking an anomaly opens granular time-series charts detailing time intervals, data sources, patient demographics, and HCP target attributes. - Candidate explanations supported by empirical proof are marked with **red squares** denoting validated evidence. - Recommended operational interventions are displayed as **blue triangles** (or blue action items). - **Closed-Loop Feedback and Export:** Users can directly interact with suggested actions—confirming adoption, rejecting steps with logged feedback, or adjusting target parameters. Marketers can subsequently download reconciled execution plans via PDF or Excel workbooks, providing field sales teams and operations planners with concrete HCP target lists and flagged patient cohorts.

[10:28](https://www.youtube.com/watch?v=Twct4OtyuZ4&t=628s) Why Pure Text-to-SQL Fails: Moving to Deterministic Engines

In the final portion of this segment, Bi contrasts COCO’s deterministic design against the prevailing enterprise AI trend of embedding conversational text-to-SQL chatbots directly into dashboards. In typical text-to-SQL implementations, a user asks a free-form question, a classification model routes the query, and an LLM-driven text-to-SQL agent attempts to query a data warehouse like Snowflake. Bi notes that this pattern encounters severe friction in regulated enterprise environments:

- **Enterprise Governance & PII Guardrails:** Unconstrained conversational interfaces risk surfacing Personally Identifiable Information (PII) or generating outputs that violate strict regulatory compliance and internal pharmaceutical policies. - **Evaluation & Query Reliability:** In mission-critical commercial analytics, SQL output must be mathematically certain. To evaluate LLM SQL reliability, Bi’s team built a rigorous benchmarking framework using a "500 golden dataset" tailored to Paxlovid across tiered difficulty levels. Tier 1 evaluated simple `SELECT` statements, while higher tiers evaluated complex joins and window aggregations. - **The "Uber Prompt" Fragility:** Even after constructing dedicated semantic layers, the engineering team found that keeping the LLM reliable demanded massive, fragile "Uber prompts" to prevent semantic drift and erroneous join logic.

Because generative SQL cannot guarantee 100% accuracy on open-ended ad hoc queries, the architecture was pivoted away from pure conversational agents toward the COCO engine. Rather than relying on real-time code generation for metric calculations, COCO utilizes deterministic KPI extraction logic in the backend, ensuring that all data points, baselines, and anomaly alerts remain mathematically reliable and auditable.

[12:31](https://www.youtube.com/watch?v=Twct4OtyuZ4&t=751s) Data Accuracy, Marketer Trust, and the 10% SQL Dilemma

While the underlying agent architecture explores diverse evidence and multiple potential explanations, handling uncertainty remains an active challenge—principally because the knowledge graph is not yet perfect. Although the team is continuously refining and improving the knowledge graph, they can guarantee that every individual metric pulled directly from Snowflake is 100% accurate. Ensuring raw numerical accuracy is the most direct pathway to winning the trust of commercial marketers.

In evaluation frameworks where agents generated SQL queries autonomously, the system achieved a 90% accuracy rate. While a 90% success rate demonstrates strong agent performance from an engineering standpoint, it leaves a problematic 10% margin of error. For commercial marketers, that remaining 10% risk of retrieving incorrect figures from Snowflake creates real friction and skepticism, making it the primary pain point the team is striving to eliminate.

```mermaid flowchart TD accTitle: Text to SQL Agent Flow and Verification subgraph Input ["Query & Context"] A["Marketer Natural Language Question"] --> B["Uber Prompt\n(Table Relationships & SQL Examples)"] end subgraph Agentic_Layer ["Agent & Execution"] B --> C["Text-to-SQL Agent"] C --> D["Generated SQL Query"] D --> E["Snowflake Data Warehouse"] end subgraph Output ["Validation & Delivery"] E --> F["Raw Verified Metrics (100% Exact)"] F --> G["Human-Readable Answer & Outlier Detection"] end ```

[13:30](https://www.youtube.com/watch?v=Twct4OtyuZ4&t=810s) Text-to-SQL Logic, "Uber Prompts", and Token Economics

An audience member asks how the agent's SQL generation logic is built. The pipeline relies heavily on what the team calls "Uber prompts," which contain detailed semantic mappings of the relationships across disparate tables alongside concrete few-shot SQL query examples (such as pre-formatted queries for calculating treatment rates). The agent ingests these Uber prompts to synthesize new SQL queries modeled after the provided templates.

However, scaling this technique introduces severe trade-offs around cost and efficiency. As more table schemas and example queries are added, the Uber prompt swells in size, consuming a massive number of tokens on every call. Even with increasingly large prompts, the approach still fails to cover 100% of the arbitrary questions a marketer might ask. Another attendee notes their own team's attempts to bridge knowledge graphs (like Neo4j) with relational SQL databases, asking how best to train models for SQL interaction. A robust semantic layer combined with prompt context is essential, but hallucinations persist at roughly a 5% rate, and pushing that error rate down to 1% or 2% remains a significant hurdle.

- **Uber Prompts**: Contain explicit table-to-table relational schemas and curated few-shot SQL examples (e.g., treatment rate calculations). - **Token Inflation**: Expanding prompt context to cover edge cases drastically increases token consumption and operating costs. - **Hallucination Margins**: SQL hallucination rates hover around 5%, requiring continuous tuning of semantic layers to reach a target of 1–2%.

[15:06](https://www.youtube.com/watch?v=Twct4OtyuZ4&t=906s) Reassuring Data Scientists: Augmentation Over Automation

The introduction of automated text-to-SQL sparked frank reactions from internal data science teams, with one colleague half-jokingly telling the speaker, "I hate your job," out of fear that the agent would replace their role. In response, the strategic focus shifted toward framing agentic AI as an assistant rather than a replacement for data science personnel.

Instead of deploying complex text-to-SQL agents that attempt to autonomously execute deep analytics from scratch, the system handles simple, predefined KPI lookups directly from Snowflake into the chat interface. For more intricate tasks, such as deep-dive cohort analysis, the agent serves as an accelerating tool designed to speed up the data scientist's workflow. This reinforces the overarching mission of the Commercial AI Acceleration team: building tools that empower existing talent rather than automating human roles away.

[16:00](https://www.youtube.com/watch?v=Twct4OtyuZ4&t=960s) Outlier Detection, Historical Lookbacks, and Predictive Cohorts

When asked by an audience member about the volume of historical data ingested and whether the system has evolved from retrospective reporting to forward-looking predictive analytics, the focus turns to the underlying time-series outlier detection framework. The engine currently ingests two years of historical data to establish baseline performance and identify meaningful deviations.

The agent actively tracks trends behind the scenes by running year-over-year (YoY) comparisons and longitudinal time-series monitoring. It also executes automated peer-to-peer cohort comparisons—such as benchmarking patients aged 15 and older against those under 15—to spot anomalies and track emerging behavioral shifts across brand and patient segments. The session concludes with an open invitation for attendees to connect for further technical discussions throughout the conference.

- **Historical Lookback**: Analyzes a rolling 2-year window of historical metrics for time-series anomaly detection. - **Trend & Outlier Mechanisms**: Leverages year-over-year comparisons and continuous time-series tracking in the background. - **Peer-to-Peer Cohorts**: Segment-based comparisons (e.g., age-bracketed cohorts) to flag statistical outliers in patient and marketer data.

## Notable quotes > [00:04](https://www.youtube.com/watch?v=Twct4OtyuZ4&t=4s) they will fire me. — Shanshan Bi

> [02:35](https://www.youtube.com/watch?v=Twct4OtyuZ4&t=155s) speed to insights problem — Shanshan Bi

> [04:42](https://www.youtube.com/watch?v=Twct4OtyuZ4&t=282s) We are not develop the COCO as the AI telling what to do next... and we still need human marketers to sign off. We are not directly replace anyone's job. — Shanshan Bi

> [10:28](https://www.youtube.com/watch?v=Twct4OtyuZ4&t=628s) 500 golden dataset — Shanshan Bi

> [10:28](https://www.youtube.com/watch?v=Twct4OtyuZ4&t=628s) Uber prompts — Shanshan Bi

> [15:06](https://www.youtube.com/watch?v=Twct4OtyuZ4&t=906s) I hate your job, — Pfizer Data Scientist

## Mentioned Shanshan Bi, Pfizer, COCO, Paxlovid, Snowflake, Neo4j, CDC, Ai4, Medicaid

## Video description Shanshan Bi - Director of Commercial AI Acceleration - AIML, Pfizer

Marketing teams generate massive amounts of performance data — but extracting timely insights still requires SQL expertise, analyst queues, and days of turnaround. COCO is an agentic AI assistant that lets marketers ask natural language questions and receive data-driven answers in seconds. Built on a multi-agent orchestration architecture, COCO routes questions through specialized AI agents that understand marketing metrics, generate and execute SQL, and deliver human-readable insights across brand performance, omnichannel engagement, and patient trends. This session covers the journey from concept to production — including the architectural decisions behind multi-agent design, lessons learned from real-world evaluation at scale, and how we're using agentic AI to democratize data access for non-technical users in a regulated enterprise environment.

[Source](https://www.youtube.com/watch?v=Twct4OtyuZ4)

中文简报

辉瑞部署Agentic AI取代营销分析排队流程 生成式text-to-SQL工具的错误率对企业决策者而言难以接受，促使辉瑞将确定性指标提取与知识图谱相结合，在数秒内诊断商业瓶颈。 为了消除延误Paxlovid等商业品牌战略调整的长达数周的临时SQL请求周期，辉瑞构建了智能体分析引擎COCO。该系统执行结构化的四阶段流程：扫描KPI以定位异常值、通过企业知识图谱生成合理的根本原因假设、跨理赔和拜访记录（detailing）数据交叉验证实证证据，并交付带有预估ROI的可行干预方案。由于治理限制、脆弱的“超级提示词”（Uber prompts）造成的Token膨胀，以及生成式text-to-SQL聊天机器人高达10%的不可接受误差率，该架构采用了后端确定性指标提取，同时保持营销人员在环（human-in-the-loop），以增强现有的数据科学工作流。 企业商业仪表板追踪数千个KPI，但仅能提供事后可见性，无法解释根本原因或指导后续操作步骤。 (0:04) 营销人员与数据科学家之间传统的临时分析流转在处理每个查询时都需要数周时间，造成了严重的洞察时效瓶颈。 (2:35) 辉瑞的COCO引擎通过四阶段流程精简分析：自动化异常检测、知识图谱假设生成、跨数据集证据收集以及行动建议。 (4:42) COCO有意采用了人机在环（human-in-the-loop）模型架构，要求人类营销人员评估并批准所建议的干预措施，而非实现决策自动化。 (4:42) 通过综合理赔、CDC数据流和拜访记录等多源数据，该系统成功确定针对医务人员（HCP）的剂量教育不足，是导致存在药物相互作用（DDI）的老年群体中Paxlovid使用率低迷的根本原因。 (6:56)

Reading summary

Pfizer developed COCO, an agentic AI analytics engine that pairs automated anomaly detection with enterprise knowledge graphs to diagnose commercial performance drops and recommend marketing actions. Presenter Shanshan Bi argues that replacing slow, iterative ad hoc query loops with deterministic data retrieval and human-in-the-loop workflows accelerates operational insights while avoiding the reliability pitfalls of generative text-to-SQL tools. The Commercial Analytics Bottleneck and Delayed Insights (0:04) Commercial brand managers tracking therapies like Paxlovid must monitor thousands of key performance indicators (KPIs) distributed across dozens of business intelligence dashboards. Bi explains that while traditional dashboards report historical performance within seconds, they cannot explain why an anomaly occurred or prescribe how brand teams should respond. When marketers observe unexpected regional adoption drops, answering follow-up questions requires manual deep dives rather than dashboard reviews. Under the conventional workflow, investigating an untreated cohort requires logging ad hoc requests with internal data science teams or external vendors. Authoring SQL pipelines, pulling extracts, analyzing cohorts, and scheduling review decks typically requires a full week per inquiry. Because initial findings often trigger secondary questions regarding root drivers, the turnaround cycle repeats iteratively. Bi emphasizes that this delay constitutes a "speed to insights problem" rather than a lack of data, often causing an entire quarter to lapse before commercial strategies can adjust to market signals. Automated Anomaly Diagnosis and Cross-Dataset Validation (4:42) To shorten the weeks-long analysis cycle, Pfizer designed COCO as a structured four-stage engine rather than an open conversational chatbot. First, the engine scans thousands of KPIs against a rolling two-year historical baseline, trailing windows of 14 to 30 days, and peer-to-peer cohorts to isolate 14 to 20 statistically significant anomalies. Second, the system queries an internal enterprise knowledge graph to retrieve multiple candidate hypotheses capable of explaining each anomaly. In the third stage, automated agents interrogate enterprise data systems to gather empirical proof, ruling out invalid hypotheses. For example, when investigating low Paxlovid initiation among patients aged 65 and older with drug-drug interaction warnings, agents evaluated claims records, patient demographics, healthcare provider (HCP) target lists, multichannel detailing logs, and public CDC wastewater surveillance data. The system ruled out data pipeline errors and identified HCP dosing adjustment uncertainty as the root cause. Fourth, COCO surfaces prioritized operational recommendations with projected return-on-investment metrics, requiring human marketers to approve, reject, or test interventions. The Operational Limits of Generative Text-to-SQL (10:28) Bi contrasts COCO's backend with prevailing enterprise trends that embed generative text-to-SQL agents into dashboards. In Pfizer's benchmark evaluations using a 500-question Paxlovid test set across varying difficulty tiers, text-to-SQL agents achieved a 90% accuracy rate. However, Bi notes that the remaining 10% error margin undermines commercial marketer trust, while open-ended conversational interfaces introduce severe governance risks around personally identifiable information (PII) and regulatory compliance. To enforce valid query logic, engineering teams developed extensive "Uber prompts" containing detailed schema relationships and few-shot SQL examples. These prompts created significant token inflation and operational costs without eliminating errors; SQL hallucination rates persisted at roughly 5%, with reducing that error margin to 1% or 2% remaining a steep hurdle. Consequently, Pfizer shifted away from live generative SQL synthesis, relying instead on deterministic backend KPI extraction to ensure metrics retrieved from Snowflake remain auditable and mathematically exact. Interface Architecture and Human-in-the-Loop Augmentation (8:32) Rather than deploying complex web applications, COCO delivers findings via structured, lightweight HTML reports organized around a macro patient conversion funnel spanning infection, treatment eligibility, Paxlovid initiation, and prescription fulfillment. The interface standardizes visual indicators: purple triangles represent detected anomalies, red squares denote empirically validated evidence, and blue triangles indicate recommended commercial actions. Marketers can examine underlying demographic charts, log feedback, and export reconciled HCP target lists to PDF or Excel workbooks. Bi notes that early text-to-SQL initiatives triggered anxiety among internal data scientists who feared automated job replacement. In response, Pfizer positioned COCO strictly as an augmentation platform. The assistant executes repetitive, predefined metric lookups and evidence collation, leaving complex strategic synthesis and final campaign sign-offs in the hands of human data scientists and marketing personnel. While COCO provides auditable, Snowflake-verified metrics and structured operational recommendations, Bi acknowledges that refining the underlying knowledge graph remains an ongoing task to resolve remaining diagnostic uncertainty. Lowering SQL hallucination rates below the current 5% threshold without incurring severe token inflation also persists as an industry-wide technical challenge.

精炼阅读摘要

辉瑞（Pfizer）商业AI加速团队开发了名为COCO的知识图谱驱动型智能体分析引擎，旨在解决制药商业营销中仪表板过度膨胀与洞察转化迟缓的瓶颈。该系统通过标准化异常检测与跨数据源归因，在人机协同框架下生成具体策略，并以确定性后端计算替代不稳定的纯Text-to-SQL架构以确保数据严谨性。 传统BI看板与沟通闭环的时效瓶颈 (0:04) 在大型跨国药企中，单一核心产品（如辉瑞的新冠抗病毒口服药Paxlovid）往往关联着分布在数十个商业仪表板中的数千个独立关键绩效指标（KPI）。演讲者Shanshan Bi指出，业务人员虽然能在数十秒内查到宏观指标，但传统商业智能工具仅能反映已发生的既成事实，无法解答“指标为何下滑”以及“后续应采取何种行动”。面对特定区域或特定未治疗人群的异动，传统看板无法提供归因与决策指引。 当营销人员尝试深入排查未预设的细分人群异动时，必须向内部数据科学团队或外部供应商提交工单。从数据工程对接、编写SQL、提取分析、制作演示幻灯片到召开汇报会议，整个闭环通常耗费至少一周；若会议中引发新的归因追问，又需要追加一周的排期分析。Bi强调，这导致决策严重滞后，往往一个季度结束商业策略仍未完成调整。企业面临的痛点并非缺乏数据或指标，而是从现有数据提炼行动方案的“洞察时效性”（Speed to Insights）极度不足。 知识图谱驱动的四阶段归因决策 (4:42) 为突破数周的数据分析延迟，辉瑞构建了基于知识图谱的洞察引擎COCO。该引擎并非开放式闲聊机器人，而是按照固定四步流水线运作的自动化分析系统：首先，专业智能体自动扫描数千个KPI在历史滑动窗口（如过去两周或30天）内的波动，筛选出14至20个具有统计学显著性的业务异常；随后，系统检索企业知识图谱提取多种潜在假设；第三阶段，智能体跨多个异构系统自动调取经验数据进行证据链比对；最终输出附带预期投资回报率（ROI）的行动方案，由人类营销人员进行最终审批、采纳、驳回或发起A/B测试。 在Paxlovid的实际业务验证中，系统捕捉到了伴有药物相互作用（DDI）禁忌警示的65岁以上患者处方启动率反常低迷的现象。知识图谱提出了三类假设：数据接口缺失或损坏、患者自身对药物配伍禁忌的担忧、以及医护人员（HCP）对合用药物下的剂量调整方案缺乏认知。COCO自主跨越医疗处方报销记录、患者画像、CDC流行病学与废水监测指标、医药代表面对面拜访纪要及机器学习模型输出进行综合交叉验算，排除了数据管道问题，证实核心阻碍在于医生缺乏针对性调量知识，从而直接为业务团队生成了更新销售代表拜访计划与精准推送剂量调整指南的行动方案。 放弃纯Text-to-SQL转向确定性计算 (10:28) Bi对比了当前企业AI领域流行的对话式Text-to-SQL看板设计，阐明该模式在制药等强监管行业存在显著缺陷。无约束的自然语言自由查询极易触犯个人身份信息（PII）泄露隐患与行业合规红线。此外，工程团队在由500个Paxlovid定制问题构成的基准黄金数据集上进行了分级测试，发现即便在大语言模型生成SQL的技术评测中取得90%的准确率，剩余10%的计算误差对商业决策者而言也是不可接受的信任硬伤。 为提升生成SQL的可靠度，团队曾构建包含复杂跨表关系及少量样本（few-shot）的“超级提示词”（Uber Prompts）。但随着数据表扩展，提示词体量激增导致调用Token成本大幅攀升，且大模型的SQL幻觉率仍停留在约5%左右，难以压降至预期的1%至2%安全区间。鉴于生成式SQL无法保证即席查询100%正确，辉瑞在底层设计上放弃了完全依赖大模型实时编写统计代码的路线，转而在COCO后台采用确定性KPI提取逻辑，确保所有指标基线与异常计算结果具备100%的数学精确度与审计可溯性。 轻量交互界面与人机协同定位 (8:32) 在前端交付层面，COCO规避了复杂的重型仪表板，采用轻量化结构HTML报告。界面顶层以患者转化宏观漏斗为锚点，依次呈现阳性感染、就医治疗、Paxlovid处方启动及药房取药脱落等阶段；界面通过视觉标记呈现核心信息：紫色三角代表漏斗转化中的统计异常，红色方块标明经数据证实的因果证据，蓝色三角则对应优先建议行动。营销人员可在线确认执行、记录驳回原因，并将带有目标医生列表与标记患者群的方案直接导出为PDF或Excel交付一线代表。 针对自动化可能引发的内部顾虑，Bi指出团队已将COCO明确界定为提升数据科学家工作流效率的辅助工具，而非替代人力的全自动接管系统，所有商业决策均必须保留人机协同（Human-in-the-Loop）审批节点。在数据纵深方面，底层时间序列异常检测机制接入了2年的滚动历史数据建立动态基线，结合同比分析（YoY）与不同年龄层（如以15岁为界）的对等分群（Peer-to-Peer Cohorts）横向比对，实现从回顾性指标监控向捕捉潜在行为变迁的演进。 COCO展示了制药企业利用知识图谱与智能体将商业归因与策略建议周期由数周压缩至近乎即时的可行性，确立了以确定性后端计算守护数据信任、人机协同保障合规的系统框架。然而，大模型SQL生成的边缘幻觉以及知识图谱的持续迭代完善，仍是该类智能体技术深化落地中需要长期权衡与攻克的关键挑战。

Detailed notes — zh

## Background 该内容由 YouTube 频道 Ai4 于 2026 年 8 月 19 日发布，主讲人为辉瑞（Pfizer）商业 AI 加速总监 Shanshan Bi。视频详细介绍了辉瑞内部自主研发的营销 Agentic AI 助手 COCO，探讨其如何通过多智能体编排与知识图谱技术打破传统 SQL 取数队列的瓶颈，秒级响应自然语言业务查询并赋能非技术营销团队。 ## TL;DR 演讲直击传统医药商业分析中指标繁多、传统仪表板无法解释“底层根因”以及跨部门提数沟通周期冗长的核心痛点，系统阐述了辉瑞自主构建的 AI 助手 COCO。COCO 依托知识图谱与时序异常检测，通过“异常发现、假设生成、多源证据链验证、决策赋能”四步闭环将海量 KPI 自动转化为可落地的行动建议。讲者结合 Paxlovid 实际案例演示了轻量化 HTML 交互报告的落地形态，并坦诚复盘了早期 Text-to-SQL 方案在合规、超级提示词维护与 10% 出错率信任危机下的技术局限，最终展示了团队如何重构确定性计算引擎并将 AI 准确定位为赋能人类专家的协同工具。 ## Atmosphere & style 整场演讲兼具幽默机智与极高的大厂实战技术密度。讲者开场以单口喜剧式的调侃迅速拉近与观众的距离，随后以冷静务实且极富同理心的视角切入药企一线业务与数据团队的协作拉锯战。在技术复盘环节，讲者毫不避讳 Text-to-SQL 方案的幻觉率与内部团队的真实阻力，现场问答互动坦诚深入，生动展现了一位资深企业级 AI 掌舵者在技术理想与业务现实博弈中打磨产品的严谨思考。 ## Key takeaways - [00:04](https://www.youtube.com/watch?v=Twct4OtyuZ4&t=4s) 医药营销分析面临海量指标过载的困境，传统仪表板只能展示“正在发生什么”，却无法回答“为什么发生”与“下一步该做什么”等业务核心问题。 - [02:35](https://www.youtube.com/watch?v=Twct4OtyuZ4&t=155s) 现行即席数据查询高度依赖跨部门沟通流转，从提数、跑流水线到开会复盘周期漫长，导致洞察生成滞后并常错失季度决策窗口期。 - [04:42](https://www.youtube.com/watch?v=Twct4OtyuZ4&t=282s) COCO 确立了四步自动化洞察流程：自动遍历 KPI 检测异常、调取知识图谱生成假设、跨库串联多源证据链排查，最终输出带预估 ROI 的行动方案。 - [04:42](https://www.youtube.com/watch?v=Twct4OtyuZ4&t=282s) 系统定位为“人机协同”的决策赋能工具，严格保留 Human-in-the-loop 机制，所有最终策略和 A/B 测试均需营销负责人签字确认。 - [06:56](https://www.youtube.com/watch?v=Twct4OtyuZ4&t=416s) 在 Paxlovid 渗透率偏低的归因中，Agent 跨越患者理赔、HCP 档案、CDC 流行病学、废水病毒监测及代表拜访等多孤岛数据形成证据闭环。 - [08:32](https://www.youtube.com/watch?v=Twct4OtyuZ4&t=512s) 产品交互摒弃复杂的重型 BI 图表，采用基于“患者旅程漏斗”的轻量化 HTML 报告，直观串联异常发现、归因假设、事实证据及行动选项。 - [10:28](https://www.youtube.com/watch?v=Twct4OtyuZ4&t=628s) 纯 Text-to-SQL 方案在受监管制药行业面临合规风控、基准评测脆弱以及维护“超级提示词（Uber Prompts）”成本过高等严重局限。 - [10:28](https://www.youtube.com/watch?v=Twct4OtyuZ4&t=628s) 辉瑞对 COCO 实施架构重构，将不稳定的动态 Text-to-SQL 转变为后端的确定性 KPI 计算引擎，以保障核心数据的绝对稳健。 - [12:31](https://www.youtube.com/watch?v=Twct4OtyuZ4&t=751s) Text-to-SQL 早期达到 90% 准确率，但剩余 10% 的出错风险与约 5% 的幻觉率使营销人员产生严重信任顾虑，促使团队追求底层数据 100% 准确。 - [15:00](https://www.youtube.com/watch?v=Twct4OtyuZ4&t=900s) 面对内部数据团队的阻力，项目明确“赋能工具而非替代岗位”的定位，标准化 KPI 自动化提取，复杂队列分析则辅助并加速数据科学家的工作流。 - [16:00](https://www.youtube.com/watch?v=Twct4OtyuZ4&t=960s) 异常检测引擎调取过去 2 年的历史时序数据，结合同比变化、长周期趋势以及特定患者队列间的横向对等比较进行多维度基线分析。 ## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=Twct4OtyuZ4&t=4s) 商业分析的现实困境：海量 KPI 与无法解答的“为什么” 讲者 Shanshan Bi 开场时幽默地调侃自己平时其实在讲单口喜剧（Stand-up Comedy），但今天绝对不会拿辉瑞（Pfizer）开玩笑，因为“要是开了玩笑他们会把我开除”。她严肃地回归主题，介绍这是辉瑞内部团队自主构建的营销 Agentic AI 助手——COCO。演讲分为三个核心板块：商业分析现存的巨大痛点、COCO 的系统架构设计与运作机理，以及最令人期待的系统实操演示。 在切入系统前，讲者向台下抛出一个直击营销本质的问题：如果你是一名医药营销人员，你如何判断某款药品的销售表现是否良好？营销策略是否真正奏效？答案显然是关键绩效指标（KPI）与各类可视化仪表板（Dashboard）。然而，辉瑞面临的残酷现实是：单单一款药物（例如新冠口服药 Paxlovid），背后就追踪着成千上万个 KPI。企业内部配备了庞大的供应商团队处理即席数据查询（ad hoc queries），也有专职的数据科学家负责解读数据并提炼洞察。但这引发了一个根本性矛盾：没有任何一个营销人员有精力每天逐一审查数以千计的指标，大家甚至根本无法完全理解每个 KPI 波动背后的真实业务含义。 当公司管理层走过来提问：“我们注意到特定地区某类患者的治疗率正在下滑，到底发生了什么？为什么会这样？”现有的所有仪表板瞬间哑火。现存的可视化工具只能被动呈现“现在正在发生什么”，却根本无法解释“为什么会发生”以及“下一步应该做什么”。而后两个追问，恰恰是管理层最为关切、决定业务生死的核心问题。 --- ### [02:35](https://www.youtube.com/watch?v=Twct4OtyuZ4&t=155s) 跨越周期的沟通内耗：从提出问题到错失良机的洞察死循环 以 Paxlovid 为例，营销人员如果想了解“过去一个季度有多少比例的符合条件患者接受了 Paxlovid 治疗”，这是一个标准问题，登录仪表板花费 30 秒就能查到准确百分比。但业务洞察往往止步于随之而来的深层追问：“为什么美国东北部特定未治疗人群的用药率偏低？他们为什么没有接受 Paxlovid 治疗？”由于现有仪表板并没有针对这种特定细分人群（Cohort）预置指标，营销人员无法从界面上直接获取答案，只能求助于数据科学家或外部技术供应商。 当前的协作流程极其缓慢且沉重：营销人员提出一个即席问题，数据科学家通常需要回复“请给我一周时间”。随后数据团队开始编写 SQL、调试数据工程流水线（DE pipeline）、从数据库中拉取数据、分析并整理出一套 PPT 汇报材料，最后再预约会议向营销团队逐页讲解。然而在会议现场，营销人员看完报告往往会立即抛出新的追问：“这背后的核心驱动因素（Key Drivers）到底是什么？”数据科学家只能再次回答：“我们需要再约一次会，回头继续深挖底层原因。” 这种跨部门沟通的拉锯战在辉瑞商业分析部门每天都在上演。等到一个问题经过反复开会、提数、复盘终于彻底查清时，整个季度可能已经悄然结束，下一季度的营销调整策略早已错失最佳窗口期。这并非数据资产匮乏的问题，企业拥有海量数据和指标，真正的死穴在于“洞察生成的转化速度”（Speed to Insights）。 --- ### [04:42](https://www.youtube.com/watch?v=Twct4OtyuZ4&t=282s) COCO 的核心理念：基于知识图谱的四步自动化洞察引擎 为了彻底打破速度瓶颈，辉瑞团队研发了 COCO——一个由知识图谱（Knowledge Graph）驱动的自动化洞察平台，其核心使命是将原始业务信号自动转化为“可直接落地执行的行动建议（Ready-to-act recommendations）”。COCO 的底层引擎主要由四步闭环构成： ```mermaid flowchart TD accTitle: COCO 自动化洞察引擎四步工作流程 subgraph S1["1. 异常信号发现"] A["遍历海量 KPI"] --> B["检测异常值 (Outliers)"] end subgraph S2["2. 假设候选生成"] B --> C["知识图谱提供多种潜在归因解释"] end subgraph S3["3. 多源证据链验证"] C --> D["自动跨数据库检索比对证据"] end subgraph S4["4. 决策赋能与人机协同"] D --> E["生成行动选项与预估 ROI"] E --> F["营销人员审核确认 / 调整决策"] end ``` 在运作过程中，AI Agent 首先会主动扫描数千个 KPI 指标，自动捕获过去两周或一个月内出现的 15 到 20 个关键异常值（Outliers），免去人工逐一肉眼排查的负担；接着，系统接入知识图谱，针对每个异常现象自动调出多个潜在的候选解释；随后，Agent 会在海量业务数据集中自动寻找支撑这些假设的事实证据，逐个排查并锁定核心驱动因素（例如特定人群——50岁以上、女性、使用 Medicaid 联邦医疗补助支付）；最后，系统输出若干项具体的行动建议供业务人员选择。 讲者特别强调，COCO 的定位绝非直接发号施令或替代人类岗位，而是提供伴随 ROI 预测的决策选项矩阵： - 营销人员享有最终决策权，可以选择采纳 A、B 方案，忽略 C 方案，或在线留下批注并触发 A/B 测试。 - 系统完整保留人类把关（Human-in-the-loop）机制，所有策略均需营销负责人最终签字确认（Sign off）。 --- ### [06:56](https://www.youtube.com/watch?v=Twct4OtyuZ4&t=416s) 实战推演：Paxlovid 高危人群用药率偏低的根因排查 讲者通过 Paxlovid 的真实业务场景展示了这套推演逻辑。业务监控发现一个特定人群的异常：65 岁以上且系统带有药物相互作用（DDI, Drug-Drug Interaction）标记的患者群体中，Paxlovid 的治疗渗透率异常偏低。COCO 启动后，首先从知识图谱中列出多条候选归因路径： 1. **数据质量问题**：是否存在数据链路丢失或理赔记录不完整； 2. **患者端顾虑**：患者是否担心正在服用的慢性病药物与新冠药物产生严重相互作用而拒服； 3. **医护人员（HCP）认知不足**：处方医生是否不清楚该药物针对此类特殊人群其实提供了剂量调整方案（Adjusted doses）。 在自动化排查并逐一排除无关假设后，Agent 锁定了“HCP 存在顾虑”这一核心驱动假设。为了验证这一假设，系统自动跨越多个独立的数据孤岛进行关联印证，调取并比对了包括患者理赔数据、HCP 档案数据、开源 CDC 流行病学数据、住院率数据、废水监测（Wastewater）病毒浓度、医药代表线下拜访记录（Face-to-face detailing）、全渠道互动记录、核心目标医生清单（Target list）以及底层机器学习模型的输出结果。在将纷繁复杂的散点数据串联成闭环证据链后，系统自动向营销团队呈现结论并附带具体的跟进策略建议。 --- ### [08:32](https://www.youtube.com/watch?v=Twct4OtyuZ4&t=512s) 产品交互范式：从复杂仪表板回归轻量化 HTML 交互报告 在演示 COCO 的前端形态时，讲者澄清该系统并非堆砌复杂 JavaScript 图表的沉重 BI 工具，而是一份条理清晰的轻量化交互式 HTML 报告。系统以“患者旅程漏斗（Patient Funnel）”为顶层框架展开呈现：从患者感染、就医确诊、开具 Paxlovid 处方，到最终是否去药房自提药物等各层级转化率切入。 ```mermaid flowchart LR accTitle: COCO HTML 交互报告视觉元素与排查路径 A["患者漏斗层级"] --> B["紫色三角形<br/>(发现异常 Findings)"] B --> C["候选归因列表<br/>(Candidates)"] C --> D["红色方块<br/>(事实证据 Evidence)"] D --> E["蓝色标识<br/>(建议行动 Actions)"] E --> F["导出 Excel / PDF<br/>(调整目标医生清单)"] ``` - **视觉符号体系**：每个漏斗层级中，系统用紫色三角形标记出检测到的异常发现（Findings），并配备时间序列走势图、数据来源区间、人群分群细节以及相关 HCP 是否在既定目标清单等元数据。 - **逐层下钻分析**：用户点击任意感兴趣的候选解释，系统即可展开下层的红色方块（事实证据 Evidence）与蓝色标识（推荐行动 Actions）。 - **即时反馈与输出闭环**：业务人员可以直接在界面上对每项行动建议标记“采纳”或“拒绝”并填写备注，最终一键导出包含患者分群画像、目标 HCP 调整名单等核心字段的 Excel 表格或 PDF 报告，快速落地到下线执行中。 --- ### [10:28](https://www.youtube.com/watch?v=Twct4OtyuZ4&t=628s) 架构反思与演进：Text-to-SQL 的技术局限与确定性引擎重构 在视频演示环节，讲者深入剖析了许多企业在尝试构建 AI 营销助手时普遍采纳的技术路径及其深层缺陷。绝大多数团队通常选择在现有仪表板上外挂一个对话 Agent：前端通过意图分类器将用户提问路由到后端，核心大脑则依赖 Text-to-SQL Agent 直接连接 Snowflake 等企业级云数仓进行动态 SQL 生成与取数。 然而，辉瑞团队在实际落地过程中发现 Text-to-SQL 存在数个难以逾越的瓶颈与巨大挑战： - **严格的安全护栏（Guardrails）与合规红线**：在高度合规的制药行业，绝对不允许 Agent 生成任何违反政策或可能泄露患者隐私（PPI）的查询内容； - **评估体系极其脆弱（Evaluation Bottleneck）**：如何确保大语言模型动态生成的 SQL 100% 正确？为了评测模型，团队甚至为 Paxlovid 单个药物构建了包含 500 个金牌基准问答对（Golden datasets）的评测框架，涵盖从基础表查询（Tier 1）到复杂多表关联等不同难度层级； - **语义层与超级提示词（Uber prompts）的维护困境**：即便团队在数据仓库之上建立了完善的语义层（Semantic layer），在实际运行中依然不得不编写极其冗长、脆弱的“超级提示词（Uber prompts）”来引导大模型写出正确的查询语句。 鉴于上述痛点，辉瑞最终对 COCO 进行了根本性的架构重构：将其从不可控的单纯 Text-to-SQL Agent 转型为后端的自动化确定性计算引擎。所有底层 KPI 的提取与计算逻辑全部转为确定性逻辑（Deterministic KPI extraction logics），从根本上保证了输出结果的高度稳健与绝对可靠。 ### [12:31](https://www.youtube.com/watch?v=Twct4OtyuZ4&t=751s) 数据准确性与 Text-to-SQL 的信任痛点 在分析推理的背后，虽然 Agent 会自主探索各种证据并生成不同维度的解释，但由于底层知识图谱（Knowledge Graph）本身并非绝对完美，系统仍面临着不确定性。为了赢得业务营销人员（Marketers）的信任，团队采取的策略是确保从 Snowflake 数据仓库中提取的每一项底层指标数据都达到 100% 的准确率。 在早期的评估框架中，直接利用 Agent 生成 SQL（Text-to-SQL）的准确率大约可以达到 90%。尽管技术团队对这一表现已经具备一定信心，但对业务端的营销人员而言，剩余 10% 的出错概率依然是巨大的隐患——因为没有人愿意在关键决策中引用来自 Snowflake 的错误数据。这种信任缺口成为了团队在产品化过程中必须重点攻克的痛点。 ```mermaid flowchart TD accTitle: Text-to-SQL 准确率与业务信任瓶颈 subgraph InputStage["自然语言查询"] A["营销人员提问"] end subgraph AgentStage["Text-to-SQL 智能体"] B["大模型生成 SQL"] C{"准确率表现 (90%)"} end subgraph DataStage["数据底座 (Snowflake)"] D["100% 正确数据提取"] E["10% 错误查询风险"] end subgraph BusinessStage["业务端反馈"] F["建立业务信任"] G["营销人员产生顾虑"] end A --> B B --> C C -->|90% 成功| D --> F C -->|10% 失败| E --> G ``` ### [13:30](https://www.youtube.com/watch?v=Twct4OtyuZ4&t=810s) 超级提示词（Uber Prompts）架构与幻觉挑战 现场观众针对 Text-to-SQL Agent 的底层构建逻辑进行了提问。演讲者解释，团队最初主要依赖一种“超级提示词”（Uber Prompts）机制。在该机制中，Prompt 会详细定义不同数据表之间的关联关系（Schema Relations），并包含丰富的少样本（Few-shot）SQL 查询示例（例如如何查询“治疗率 / Treatments Rate”等特定营销指标）。当用户提出问题时，Agent 结合这些详尽的上下文与示例生成全新的 SQL 查询。 然而，这种做法很快遭遇了瓶颈与成本压力。随着业务场景越来越复杂，超级提示词变得越来越庞大，单次交互消耗的 Token 数量激增，却依然无法覆盖营销人员提出的所有边缘问题。在与台下正在尝试使用 Neo4j 知识图谱结合 SQL 数据库构建类似系统的同行交流时，演讲者指出，语义层（Semantic Layer）与 Prompt 设计对于保证 SQL 的正确性至关重要，但模型仍有约 5% 的幻觉率（Hallucination Rate）。尽管团队极力尝试将其压缩至 1% 到 2%，这一技术差距依然存在。 - **超级提示词（Uber Prompts）构成**：包含多表关联架构、业务字段映射以及多场景少样本 SQL 模板。 - **主要瓶颈**：Prompt 膨胀导致 Token 消耗急剧攀升，但依然难以实现 100% 的查询意图全覆盖。 - **幻觉率现状**：生成式 SQL 的幻觉率目前维持在 5% 左右，团队正持续优化以争取压降至 1%~2%。 ### [15:00](https://www.youtube.com/watch?v=Twct4OtyuZ4&t=900s) 赋能而非替代：AI 助手与数据科学家的协同定位 在推进 Text-to-SQL 的过程中，团队还遭遇了来自内部数据团队的微妙阻力。有数据科学家曾半开玩笑地直言："I hate your job"（“我讨厌你的工作”），因为这看起来像是在试图用 AI 取代数据科学家的岗位。这一反馈促使团队重新审视 Agentic AI 在企业中的落地路径与定位：对于已经在 Snowflake 中计算好的标准化 KPI，Agent 只需执行简单的点选并直接填充至对话框即可；而对于高度复杂的队列分析（Cohort Analysis）或多维下钻，则不应让 AI 去强行替代人类写复杂的 SQL。 团队最终将 AI 的定位调整为“赋能工具”而非“替代者”，并专门设计了人群队列分析等辅助工具，以加速数据科学家的日常工作流。演讲者强调，商业 AI 加速顾问团队（Commercial AI Acceleration Advisor）的核心愿景始终是 "We're building tools. We're not replace anyone's job"（“我们在打造工具，而不是取代任何人的工作”）。 ### [16:00](https://www.youtube.com/watch?v=Twct4OtyuZ4&t=960s) 历史数据跨度与时序异常检测分析 在最后的问答环节中，观众询问了系统所依赖的历史数据跨度以及预测性能力。演讲者介绍，针对核心业务 KPI 的异常检测（Outlier Detection），系统底层通常会调用过去 2 年的历史数据进行时序分析与基线对比。 除了回顾性数据查询，系统还具备深入趋势洞察的能力。在后台运作的异常检测引擎中，融合了同比分析（Year-over-Year Tracking）、时序趋势追踪以及同业/细分人群对等比较（Peer-to-Peer Comparisons，例如针对 15 岁以上与 15 岁以下特定患者队列的对比）。通过多维度的基线比对，AI 能够敏锐捕捉营销指标中的异常波动，为后续预测与策略调整提供坚实支撑。 - **数据时间跨度**：通常拉取过去 2 年的时序数据用于构建基准线与异常检测。 - **多维比对机制**：集成 YoY 同比变化追踪、长周期时序趋势以及特定人群队列（如 Cohort > 15 与 Cohort < 15）之间的横向对等比较。 ## Notable quotes > [00:04](https://www.youtube.com/watch?v=Twct4OtyuZ4&t=4s) 要是开了玩笑他们会把我开除 — Shanshan Bi > [00:04](https://www.youtube.com/watch?v=Twct4OtyuZ4&t=4s) 我们注意到特定地区某类患者的治疗率正在下滑，到底发生了什么？为什么会这样？ — 管理层 > [02:35](https://www.youtube.com/watch?v=Twct4OtyuZ4&t=155s) 为什么美国东北部特定未治疗人群的用药率偏低？他们为什么没有接受 Paxlovid 治疗？ — 营销人员 > [02:35](https://www.youtube.com/watch?v=Twct4OtyuZ4&t=155s) 请给我一周时间 — 数据科学家 > [02:35](https://www.youtube.com/watch?v=Twct4OtyuZ4&t=155s) 这背后的核心驱动因素到底是什么？ — 营销人员 > [02:35](https://www.youtube.com/watch?v=Twct4OtyuZ4&t=155s) 我们需要再约一次会，回头继续深挖底层原因。 — 数据科学家 > [15:00](https://www.youtube.com/watch?v=Twct4OtyuZ4&t=900s) I hate your job — 数据科学家 > [15:00](https://www.youtube.com/watch?v=Twct4OtyuZ4&t=900s) We're building tools. We're not replace anyone's job — Shanshan Bi ## Mentioned Shanshan Bi, Pfizer (辉瑞), COCO, Paxlovid, Snowflake, Neo4j, CDC (美国疾病控制与预防中心), Medicaid, Ai4 ## Video description Shanshan Bi - Director of Commercial AI Acceleration - AIML, Pfizer Marketing teams generate massive amounts of performance data — but extracting timely insights still requires SQL expertise, analyst queues, and days of turnaround. COCO is an agentic AI assistant that lets marketers ask natural language questions and receive data-driven answers in seconds. Built on a multi-agent orchestration architecture, COCO routes questions through specialized AI agents that understand marketing metrics, generate and execute SQL, and deliver human-readable insights across brand performance, omnichannel engagement, and patient trends. This session covers the journey from concept to production — including the architectural decisions behind multi-agent design, lessons learned from real-world evaluation at scale, and how we're using agentic AI to democratize data access for non-technical users in a regulated enterprise environment. [Source](https://www.youtube.com/watch?v=Twct4OtyuZ4)

Source quotes

[0:04] Shanshan Bi: 要是开了玩笑他们会把我开除 [0:04] 管理层: 我们注意到特定地区某类患者的治疗率正在下滑，到底发生了什么？为什么会这样？ [2:35] 营销人员: 为什么美国东北部特定未治疗人群的用药率偏低？他们为什么没有接受 Paxlovid 治疗？ [2:35] 数据科学家: 请给我一周时间 [2:35] 营销人员: 这背后的核心驱动因素到底是什么？ [2:35] 数据科学家: 我们需要再约一次会，回头继续深挖底层原因。 [15:00] 数据科学家: I hate your job [15:00] Shanshan Bi: We're building tools. We're not replace anyone's job

Source quotes

[0:04] Shanshan Bi: they will fire me. [2:35] Shanshan Bi: speed to insights problem [4:42] Shanshan Bi: We are not develop the COCO as the AI telling what to do next... and we still need human marketers to sign off. We are not directly replace anyone's job. [10:28] Shanshan Bi: 500 golden dataset [10:28] Shanshan Bi: Uber prompts [15:06] Pfizer Data Scientist: I hate your job,

Original transcript

[0:04] My name is Shen Shen. I'm actually  doing the stand-up comedy. So, [gasps] [0:12] but today I'm not making joke  on Pfizer cuz they will fire me. [0:19] Yeah. So, so instead I will talk about our agentic  AI assistant for marketing we built this in [0:27] at Pfizer internally. This is not a joke.  So, my talk is simple includes three parts. [0:34] First of all, I will talk about what happens now  with the pain point our commercial analytics has. [0:40] And we do have a lot of big headaches over there.  And seconds and we will talk about Coco design. [0:46] So, why we want to have a Coco and what the  benefits we can get it from Coco, how Coco works. [0:52] And the last part is my favorite is a demo for  the Coco. So, before getting started, I have a [0:59] questions for you. So, if you are the marketer,  how could you know our drug selling performance [1:06] is good is good? And how do you know our market  strategy is working? Yeah, perfect. So, the KPIs. [1:15] And we use the dashboard to visualize KPIs. But my  question is how many KPIs do you have? Yeah. So, [1:23] So, actually for each of the drugs at Pfizer, for  example, Paxlovid, which is the COVID treatments, [1:29] they have a thousands of KPIs. And we have so many  dashboard. And we have a vendor team supporting [1:36] us get ad hoc access. We have an internal data  scientist to explain and generate the insights [1:42] from the data, right? So, but do you have time to  go through the every single KPIs every single day? [1:50] Or even for a dashboard, you just review it and  we have so many KPIs. And how could you know what [1:56] happens behind the KPIs? What's the meaning of the  KPIs? You probably don't understand everything. [2:02] So when the leadership ask you question, oh, we  saw that the treatments rate for the specific [2:07] regions is going low. What's happens and why  that happens? And dashboard won't answer any [2:15] answer any questions you have or leadership have.  So the dashboard is telling you what happens now, [2:21] what is happening, but it's no one's telling  you why that happening and what to do next. [2:26] So but this two follow-up question  is the concerns of the leadership. [2:31] It's what we want to answer using our genetic AI. [2:35] So for example, take the Paxlovid as an  example and usually the market marketers [2:41] will have a question, what's the percentage of the  eligible patients were treated by Paxlovid in the [2:46] past a quarter? So it's easy, it's simple ones.  You can go to the dashboard and find the results. [2:52] Probably you need the 30 seconds to go through  that and find the exactly numbers percentage. [2:58] But you probably wonder of what is a specific  cohort analysis like northeast like on treatment [3:06] patients, what they what what happens there?  So why they didn't get treated with a Paxlovid, [3:12] right? So for this follow-up questions,  the dashboard doesn't have a KPI populated [3:18] and you cannot get the numbers from the  dashboard. And what you do, you're going to [3:22] go to data scientist or vendor support. So that's  what happens now and our marketers ask a questions [3:31] and our data scientist said, oh, give me a one  week. I will reach out to our DE pipelines to get [3:36] the sequel query and pull the data out and then  we will generate some insights from the data sets [3:42] and make a deck and we will schedule a meeting  to go through the deck and answer questions. [3:47] And but the when the marketers saw the desk,  they have another follow-up questions. So, what [3:53] happens? What's the key drivers behind the scenes?  Right. So, and the scientist said that I will [3:58] going to set up another meeting later. And we will  dig into that was the reasons behind the scenes. [4:04] And this happens every single day at Pfizer  at the commercial analytics is this process [4:10] over and over again? It's pretty slow, right?  So, when when you get the all the things got [4:17] things answered, and probably the quarter  is over. So, what about your next quarter [4:23] marketing strategy? Are you going to make the  improvement, make adjustment? You don't know. [4:28] So, is it the data problem? It's not. We  have a data there. We have a KPI there. [4:33] But the things is the speed. The speed to  insights problem. So, that's why we built a Coco, [4:42] which is a knowledge graph powered insights  platforms, can turns the signals to the ready [4:50] to act recommendations automatically. So, the Coco  engines will do four things. The first of all, [4:57] the agents going to go through thousands  of KPIs for you and try to find outliers. [5:04] And you don't have to go through that by yourself.  You don't have time to do it, right? So, we find [5:08] the outliers, probably 14, 15, 20 outliers find it  for the past the 2 weeks or past a month. And then [5:18] we will have the knowledge graph tell the agents,  oh, we have the potential candidates explanations [5:25] for those outliers. And we don't just have one. We  have options, we have several candidates. And our [5:32] agents going to keep going through the each of the  candidates and find evidence from our data sets. [5:39] So, that's also the automatic process to find  the potential evidence from our data sets. For [5:46] examples, we have uh the low treatment rate.  We have that for specific uh patient cohort, [5:53] like a patient over 50 is female using Medicaid to  to pay their prescriptions or something like that. [6:00] We will go through each of them trying trying  to find the key drivers for this outliers. [6:04] And the final uh the finally, we will  get give you some actions options. [6:11] So, you can make the decisions. We want to  do ABC or we want to do AB or just ignore C. [6:17] You can leave the comments. You can confirm  it. We're going to do this next uh quarter [6:21] and we're going to reject it because that's not  correct. Or we can leave the notes. Probably [6:25] that's the reasons we want to try it out. We  want to do the AB testing or something like that. [6:30] So, we are not uh develop the Cocoa as the AI  telling what to do next. We'll just give you [6:37] the options and the recommendations. This  could potentially works and also include [6:42] ROIs for each of the actions for your reference.  And we still need human marketers to sign off. [6:49] We are not directly anyone replace anyone's job  and we still need a human make the decision. [6:56] So, let's take examples of a Paxlovid and we have  the uh specific cohort is patients over 65 and [7:03] have a DDI reasons flagged. And why the treatment  is so low for that specific cohort? And we find [7:11] that the potential explanations could be data  quality issues. This This happens and then we have [7:17] data quality issues that the data is not complete.  And we have the concerns from patients, I'm taking [7:23] uh this drug or I cannot take Paxlovid because I'm  afraid they're going to interactions each other. [7:29] And we probably the HCP doesn't know and we have  the COVID uh we have concerns about using COVID [7:36] cuz we they don't know so we have uh adjusted  doses of the COVID of the Paxlovid to treat [7:41] the specific cohort of the patient. So, there's so  many reasons and we we find out we rule out a lot [7:48] of the candidates and we find out oh, HCP have  a concerns. And if the HCP have the concerns, [7:54] is there any evidence from our data that sets to  support that assumptions? And we go through the [8:01] all the cohort analysis detailing and we have  a claim data, patient data, HCP data, we have [8:06] open source CDC data, hospitalizing data with the  water data, we have the face-to-face detailing, we [8:12] have omni channel, we have a target list, we have  machine learning model output, so many things. We [8:18] connect those data sets and the outcomes together  and try to find evidence for you for your review. [8:25] And the final final things we give the  actions. And the marketers make the decision. [8:32] So, this is a demo for the cocoa. And in this uh  in this dashboard, it's not a dashboard, it's not [8:39] JavaScript, it's just a HTML report. So, in this  HTML report, we will start with a patient funnel. [8:46] So, what the what is a patient funnel? It's how  many patients get infected, how many patients [8:51] get treatment, how many patients get treatment  by Paxlovid, how many how many patients didn't [8:56] pick up their prescription, something like that.  We have that information, we start from there. [9:01] And we start explore and we'll see for  each of the layers, we have the findings [9:07] which is the outliers. And we have the findings  as a triangle, purple triangles and for each [9:13] of the findings, we have a time series plot,  those detail informations, data sources we use, [9:19] what the time periods we use and what's the  cohorts were different cohorts for patients. [9:25] And is the HCP on the target list or something  like that. We have that informations. [9:30] And for each of the findings, we'll have  a candidate explanation. So, you have [9:34] so many options, you can choose which ways  you are interested in, you want to dig into, [9:39] you just click the buttons and they  will show the details like the evidence. [9:44] Like the red square is the evidence and we have  actions, that's a blue square, a blue triangle. [9:51] So, the people can give us the feedbacks for  each of the actions. We want to take it or we [9:56] don't want to take it and we leave the notes there  and finally we download the PDF, the Excel sheet [10:03] from this report and we will know that we truly  uh adjust the HCP target list or which outliers [10:10] are not flagged that this HCP is which outliers  we flagged the patient cohort, something like [10:15] that. We include all the information in that  Excel sheet. We're simple and straightforward. [10:21] This is the video and can we play the video? Yeah. [10:28] So, we have the actual functionality available  in this in this video. So, you'll probably think [10:34] uh a lot of companies I know, they have a  dashboard, they have a agents in the dashboard. [10:40] So, if there's a no the KPIs is not available  in this in the dashboard, you can ask the agents [10:46] and then probably in the back ends, you have  some contents classifiers to routing your [10:51] questions to different places and also you have  the brains called agent card text to SQL agent. [10:58] So, you build the connections between your agents  to the snowflakes and you want to query that data [11:03] for your specific question, but the primary  challenge is first of all, the guardrails. [11:10] We cannot like ask the agents to populate anything  which is the PPI against any like a policies, [11:17] right? And another big headache is the evaluation.  How could you know the SQL queries generated by [11:24] by agent is 100% correct? So, you are probably  build evaluation frameworks like I built the [11:32] evaluation frameworks. We have the 500 golden data  sets for specific drugs, for example, Paxlovid. [11:40] And we have the different difficulty layers, like  the the layer one the tier one is maybe just a [11:48] very simple SQL agent select something from the  table. And we have a another like a complicated [11:54] SQL generated. But I still find the challenge is  you have a semantic layer build ups for your data [12:01] sets, but you still have a Uber prompts to guide  the agents to make the correct SQL queries to [12:07] query your data out. So, that's a challenge that  I face. So, that's why we built this Coco agent is [12:15] is engine is not agent anymore. So, this engines  in the back end all the KPIs is populate very [12:23] uh very solid, like reliable. So, we use a  deterministic KPI extractions logics behind [12:31] the scenes. But we still have uncertainties. We  will have our agent explore different evidence, [12:38] different explanations. There as uncertainties  happens over there is because our knowledge graph [12:44] is not perfect. So, we can improve the knowledge  graph, but we can guarantee the numbers every [12:49] single numbers we pull out from the Snowflake  is 100% correct. So, that's the easier way to [12:57] earn the trust from our marketers. Cuz I have  experience for the evaluation frameworks we [13:02] use the agents to generate the SQL, and our  accuracy could reach 90%. So, we can Yeah, we [13:10] we probably confidence about what the  agent performance, but the marketers [13:15] still can't have the concerns about another 10%  which is could get the wrong numbers from SQL [13:20] from Snowflake. So, that's the pinpoints  we want to solve. And this is my time. [13:30] So, how are you building the agent SQL logic? How  does that work? Oh, text-to-SQL agent? Yes. Yeah. [13:38] So, for text-to-SQL agents, we have some Uber  prompts, which is include the relations between [13:45] these two different tables, and we have examples  of the SQL queries. For example, we want to [13:50] query the treatments rate, and we will include the  examples of the SQL queries in the Uber prompts, [13:55] and then we will feed the Uber prompts to the  agents and to generate the new SQL queries based [14:02] on that example. So, that we are doing. But,  the things is I find the challenges the cost [14:07] cuz our Uber prompts is getting bigger and  bigger. So, they burn a lot of tokens over there. [14:14] So, but it is still cannot cover 100% questions  the marketer has. So, that's pretty challenging [14:21] part. You're You're using the text-to-SQL agent?  No? Or you want to build it? Uh we're working on [14:29] uh building one now with using knowledge graphs,  and one of the things that I've been going through [14:33] is how are we going to actually interact with the  knowledge graph? Mhm. And uh there's there's been [14:36] a couple different ones like using Neo4j, but then  there's also if we did the SQL database, how do we [14:41] actually get it to where we can train the AI to  to do the SQL queries? Yeah, I think Uber prompts [14:48] and the semantic layers are very important  to get the SQL queries correct. But, I still [14:53] find some hallucination rate is probably high.  It's a 5%. I try to reduce that to the 2% or 1%, [15:00] but it's still, you know, have a gaps over there.  So, when I build the text-to-SQL queries, and [15:06] then the data scientists come to me, I hate your  job. So, because you are trying to replace a data [15:11] scientist's job. But, could we use the agentic AI  in other ways? We don't do text-to-SQL anymore. We [15:18] probably do the very simple one. So, if you have  a KPIs populate in your snowflakes, and you can [15:23] just select that numbers and populate in the chat  box, that's fine. But if you're the complicated [15:29] like cohort analysis or detailed digging into,  could you suggest in different ways like we [15:34] speed up the process that to help data scientists  to go through the process instead of replace their [15:41] jobs to generate SQL. Because generating SQL is  a data scientist job, and we don't replace them. [15:47] We're helping them. So, we design a cohort as  a tool. So, that's also our visions of our AI [15:53] acceleration advisor. We're building tools. We're  not replace anyone's job. Quick question for you. [16:00] Over here. I'm right here. I'm over here on the  side. Okay. All right. Quick question for you. [16:06] Um one, how much how many years of historical  data are you using? Number one, and number two, [16:14] Oh, could you repeat the number one, sorry? So,  how much how much historical data are you using? [16:20] Like is it Marketers for for this one? Right. Look  like it was about a year. I saw 2025 and 2026. So, [16:28] look like you had historical data that you were  pulling the information or the insights from. [16:35] Uh How many how many marketers using or are  you many years of data? Years of data? Yeah, [16:41] with for the KPIs, we do have time series outlier  detections objects as well. So, I usually cover [16:48] the past two years to detect the outliers. And  the reason why I'm asking that question is because [16:54] have you guys gotten to the point where it  becomes more predictive where you can actually [16:59] I want to say see forward, right? Based on past  trends that you see. Yeah, we do consider trends. [17:07] Because when I do the outlier detections,  we have a year-to-year comparison, and also [17:11] time series tracking. Those features is directed  behind the scenes as well. And also compared the [17:17] like the peer-to-peer comparisons. For example,  for the cohort above 15 and the cohort under 15. [17:24] So, we compare those pairs and try to detect the  outliers. We keep eyes on it. Yeah. Okay, cool. [17:30] So, we're on time. So, if you have questions, feel  I'm pretty open this afternoon. Thank you so much.