Bianco Warns 85% of AI Initiatives Fail on ROI From Unclear Goals

Shoot for the Target, Not the Stars With Fancy Quant — Ai4

Ai4

Enterprise AI projects collapse primarily from undefined success metrics, diffused ownership, and poor data quality rather than flawed mathematical modeling.

AI-assisted summary of Shoot for the Target, Not the Stars With Fancy Quant — original source. Check the original for full context.

Source published: 2025-08-26

Detailed notes

## Background Published on YouTube by Ai4 on August 26, 2025, this talk features Dimitri Bianco, Quant Founder at Fancy Quant. The session explores why organizations overspend on AI and machine learning initiatives with disappointing returns, outlining strategies to achieve small wins and establish modeling as a permanent core competency.

## TL;DR Dimitri Bianco examines why 70% to 85% of enterprise AI and machine learning initiatives fail to achieve positive ROI, attributing these breakdowns to flawed planning, communication chasms, inadequate data quality, and skill mismatches. He warns against multi-year monolithic automation traps and hollow vendor promises, urging leaders instead to target high-leverage, bite-sized wins such as using LLMs for regulatory model documentation. To succeed, organizations must establish single-threaded ownership across the entire modeling pipeline, respect the foundational dimensions of data quality, and invest in continuous employee education. Ultimately, transforming AI into a sustainable business driver requires a disciplined 'crawl, walk, jog, run' progression backed by technical rigor, validation governance, and clear business alignment.

## Atmosphere & style The presentation carries a pragmatic, refreshingly candid tone delivered with the hard-won perspective of an experienced quantitative practitioner cutting through modern corporate hype. Bianco uses sharp, relatable war stories—from exposing vendor consulting traps to recounting textbook-laden backpacks—to ground theoretical management concepts in real-world friction. The talk feels like an urgent yet constructive wake-up call, balancing frank critiques of organizational dysfunctions with methodical, empowering guidance for practitioners and executives alike.

## Key takeaways - [00:04](https://www.youtube.com/watch?v=0xC6tmtyDr8&t=4s) AI and machine learning initiatives face a staggering 70% to 85% failure rate, primarily driven not by technical non-completion but by poor economic ROI, budget blowouts, and schedule overruns. - [00:04](https://www.youtube.com/watch?v=0xC6tmtyDr8&t=4s) Chasing the latest LLM hype for every enterprise use case is counterproductive; successful teams match the right quantitative and AI tools to specific business problems. - [01:55](https://www.youtube.com/watch?v=0xC6tmtyDr8&t=115s) Planning collapses when executives issue vague mandates without defining success metrics upfront and when teams underestimate the triad of time, internal skills, and infrastructure. - [04:12](https://www.youtube.com/watch?v=0xC6tmtyDr8&t=252s) Organizations achieve higher ROI by decomposing workflows into bite-sized, high-leverage tasks (such as automating compliance dossiers) rather than pursuing monolithic, multi-year model automation moonshots. - [06:03](https://www.youtube.com/watch?v=0xC6tmtyDr8&t=363s) Enterprise AI initiatives frequently 'die quietly' across departmental silos when single-threaded ownership is absent and shifting dependencies slip unmonitored. - [07:42](https://www.youtube.com/watch?v=0xC6tmtyDr8&t=462s) Beware of vendor software traps that disguise expensive consulting engagements as turnkey tools, and never build models in an engineering silo without operational and end-user stakeholders. - [09:45](https://www.youtube.com/watch?v=0xC6tmtyDr8&t=585s) Communication breakdowns are symmetrical: executives struggle to provide actionable business context, while technical teams often overwhelm stakeholders with mathematical jargon rather than validating core assumptions. - [12:11](https://www.youtube.com/watch?v=0xC6tmtyDr8&t=731s) Conflicting internal definitions of deadlines and punitive blame cultures for technical setbacks destroy team morale and derail long-term AI capabilities. - [13:36](https://www.youtube.com/watch?v=0xC6tmtyDr8&t=816s) Standard software methodologies like Agile often create friction when applied strictly to iterative, hypothesis-driven quantitative research. - [14:14](https://www.youtube.com/watch?v=0xC6tmtyDr8&t=854s) Massive data volume cannot compensate for poor quality or lack of predictive signal; advanced algorithms and synthetic data techniques cannot replace foundational data readiness. - [16:09](https://www.youtube.com/watch?v=0xC6tmtyDr8&t=969s) Retaining top-tier quantitative and AI talent requires providing intellectually engaging work and dedicated working hours for continuous learning in a fast-evolving field. - [17:46](https://www.youtube.com/watch?v=0xC6tmtyDr8&t=1066s) Sustainable AI ROI follows a disciplined 'crawl, walk, jog, run' progression that embeds modeling, validation governance, and business alignment as permanent core competencies.

## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=0xC6tmtyDr8&t=4s) The Reality of AI Project Failures and LLM Hype

Dimitri Bianco opens by introducing his background in quantitative finance, framing the term "quant" for the audience. In Wall Street lore and Hollywood depictions, quants are the individuals brought into trading floors to build the underlying mathematics, statistical tests, and models. In modern tech parlance, that translates directly to machine learning engineers, AI engineers, and quantitative researchers. While the broader tech and conference circuit remains fixated on large language models (LLMs), Bianco urges attendees to take a step back and recognize that artificial intelligence spans a vastly wider landscape of tools. Chasing LLMs for every enterprise problem is akin to relentlessly "hammering the square peg" into a round hole, inspiring the session's theme: "Shoot for the target, not for the stars."

```mermaid flowchart TD accTitle: Tech vs AI Project Failure Rates and Root Causes subgraph Rates["Failure Rates"] TP["Standard Tech Projects 25% – 50% Failure"] AP["AI / ML Projects 70% – 85% Failure"] end

subgraph Drivers["Core Drivers of Failure"] P["Poor Planning & Vague Goals"] O["Diffused / Missing Ownership"] C["Two-Way Communication Breakdown"] R["Resource & Skill Misalignment"] end

Rates --> Drivers ```

While standard software and enterprise tech projects already carry a high failure rate of 25% to 50%, AI initiatives experience an alarming failure rate between 70% and 85%. Crucially, Bianco clarifies that "failure" does not simply mean a team abandoned a project before crossing the finish line. Rather, it is fundamentally a failure of return on investment (ROI). Delivering a model or pipeline that had a $500,000 budget and a set deadline by spending $2 million is an operational completion, but an economic catastrophe. These breakdowns consistently trace back to four interconnected fundamentals: planning, communication, data, and skills.

- Typical enterprise technology project failure rate: 25% to 50%. - AI and machine learning initiative failure rate: 70% to 85% (measured primarily by poor ROI and massive cost/schedule overruns). - The four recurring failure pillars: planning, communication, data, and skills.

---

[01:55](https://www.youtube.com/watch?v=0xC6tmtyDr8&t=115s) Planning Pitfalls: Vague Goals, Vanishing Ownership, and Resource Miscalculations

Proper planning is the foundation of project viability, yet executives frequently return from industry events dazzled by emerging tech, demanding that teams immediately "build me AI" without identifying a coherent business problem or a measurable use case. Even when models are successfully trained and deployed to production, business stakeholders often gripe that the project fell short simply because nobody bothered to define success metrics at the outset. When expectations are not anchored to clear numbers from day one, even technically sound systems are branded as failures.

```mermaid flowchart LR accTitle: The Resource Triad for AI Initiatives subgraph Resources["The Resource Triad"] direction TB T["Time Almost universally underestimated"] P["People In-house skill gaps & learning curve"] Tech["Technology Infrastructure & vendor patchwork"] end

Requirements["Vague Executive Mandate"] --> Resources Resources --> Risk["Slipped Deadlines & Scope Creep"] ```

A major contributor to planning collapse is the failure to properly map the resource triad: time, people, and technology. Bianco highlights that teams almost universally misjudge the time component of an AI roadmap because unforeseen technical hurdles, data hygiene issues, and infrastructure roadblocks inevitably surface. Furthermore, organizations frequently fall victim to vendor sales pitches promising plug-and-play AI platforms that will magically solve every enterprise issue, only to discover midway through execution that they lack both the appropriate data architecture and the internal personnel required to operationalize the software.

- Success definitions must be established prior to development; hitting technical milestones means nothing if the business unit had unspoken, unmeasured expectations. - Project ownership must clearly delineate who owns the delivery, who owns the success metrics, and who owns the downstream business integration. - The time required for model delivery is almost guaranteed to be underestimated due to hidden technical and infrastructural hurdles.

---

[04:12](https://www.youtube.com/watch?v=0xC6tmtyDr8&t=252s) The Multi-Year Automation Trap vs. Bite-Sized Wins

To illustrate the danger of over-ambitious planning, Bianco shares a cautionary war story from a major financial firm. Long before the recent explosion of generative AI, leadership launched a massive initiative intended to completely automate the quantitative model development lifecycle and eliminate the need for manual development workflows. The organization spent five to seven years and fortunes of capital pursuing this grand vision. The endeavor ended abruptly when a incoming CEO reviewed the astronomical spend, saw negligible return, and grew furious at the squandered resources. The fundamental error was that executive leadership treated complex quantitative modeling as a trivial, single-step procedure rather than an intricate, iterative discipline involving hypothesis testing, high-dimensional statistical exploration, and continuous validation.

```mermaid flowchart LR accTitle: Monolithic Automation Trap vs. Bite-Sized Documentation Win subgraph Failed["The Flawed Giant Bet"] M1["Automate Full Model Development"] --> F1["5–7 Years & Fortunes Spent"] --> F2["Executive Backlash & Cancellation"] end

subgraph Successful["Bite-Sized Strategic Win"] S1["Quant Writes 1,000–2,000 Lines of Code"] --> S2["LLM Formats Scrappy Notes to Template"] --> S3["Generates 70–80 Page Regulated Dossier"] end ```

In contrast to monolithic moonshots, Bianco champions decomposing problems into bite-sized, high-leverage workflows. For example, quants at major banks routinely write 1,000 to 2,000 lines of complex code, perform statistical distributions checks, and navigate high dimensionality, but they are also burdened with regulatory documentation. Because banking is an intensely regulated industry, a single model dossier can run 70 to 80 pages long. Forward-thinking institutions now task LLMs with ingesting an engineer's running notes and draft outputs, reformatting that information directly into compliance-approved templates for human review. Choosing targeted operational friction points over sweeping, vague aspirations like "improve customer experience" produces tangible, high-ROI outcomes.

- Monolithic planning failure: A multi-million-dollar, 5-to-7-year initiative aimed at automating entire model development workflows was shut down after failing to recognize the real-world complexity of the discipline. - Practical application: Highly regulated banks require 70 to 80 pages of model documentation; using LLMs to convert code annotations into standard reporting templates automates a massive operational bottleneck. - Broad corporate slogans like "improve customer experience" lack mechanistic problem definitions and cannot serve as actionable engineering targets.

---

[06:03](https://www.youtube.com/watch?v=0xC6tmtyDr8&t=363s) The Silent Death of Projects and Diffused Accountability

When ownership is left unassigned, even minor project frictions rapidly devolve into interdepartmental feuding. When a model stalls or data lags, teams reflexively point fingers: data engineering blames IT infrastructure, quants blame low-quality input data, and business leaders blame the technical teams for missing delivery dates. These disputes arise not from insurmountable technical obstacles, but because no single party is held accountable for tracking milestones and driving cross-functional alignment. The resulting administrative friction burns out staff, spawns endless status meetings, and wastes months of corporate investment.

Bianco points out that an alarming number of enterprise AI projects do not fail with a bang—they simply "die quietly." An enthusiastic leader rallies a technical team around an AI concept, but progress halts when the data department estimates a three-to-four-month turnaround to provide the necessary tables. As corporate priorities inevitably pivot, the data team sidelines the ticket, and the quantitative team deprioritizes the model while waiting in limbo. Months later, during an executive or board review, leadership asks for an update on the flagship initiative, leaving managers scrambling to explain why the project evaporated without anyone formally raising an alarm.

```mermaid flowchart TD accTitle: How Projects Die Quietly Across Silos A["Lead Gets Excited & Launches AI Initiative"] --> B["Data Team Requests 3–4 Months for Pipeline"] B --> C["Corporate Priorities Shift"] C --> D["Data Team Silently Drops Extraction"] D --> E["Quant Team Shelves Model Indefinitely"] E --> F["Board Inquires on Delivery Status Months Later"] F --> G["Mad Scramble to Justify Project Disappearance"] ```

- Diffused ownership breeds inter-departmental blame games over data quality, computing power, and scheduling delays. - "Quiet project deaths" occur when dependency timelines stretch out, secondary teams shift focus, and nobody retains explicit responsibility to push deliverables across organizational silos. - Rigorous project management and transparent single-threaded ownership are essential to prevent initiatives from languishing in limbo.

---

[07:42](https://www.youtube.com/watch?v=0xC6tmtyDr8&t=462s) Vendor Tool Traps, Self-Learning Curves, and Departmental Blind Spots

Walking through conference exhibition floors reveals vendor booths uniformly claiming their proprietary AI software can resolve any corporate ailment. Bianco recounts a telling conversation with a vendor rep: after explaining he was an independent practitioner without a company to sell to, the vendor insisted they could still help him. When pressed on what their product actually did, the vendor simply answered, "We do AI." When Bianco asked what that meant in practice, the vendor explained that they would come in, review his operations and data, identify his business problems, and then build a bespoke system—an engagement Bianco observes is just an extraordinarily expensive consulting assignment disguised as enterprise software.

Attempting to assemble modern architectures by stitching together best-of-breed vendors—combining disparate platforms for data management, cloud infrastructure, and project tracking—often results in an unmaintainable patchwork. Bianco emphasizes that external tooling must be balanced against the internal learning curve. When foundational models rapidly shift (such as moving through successive generations of frontier models like ChatGPT), organizations cannot assume their developers will assimilate the new paradigms overnight. Furthermore, building in an engineering vacuum without operational stakeholders is fatal; bringing an operations team into the room only after a model is built inevitably surfaces practical reasons why the tool cannot be deployed in production.

- Beware of ambiguous vendor claims that masquerade expensive, open-ended consulting engagements as turnkey software platforms. - Complex vendor stacks create severe integration headaches when organizations attempt to patch disparate infrastructure, data, and tracking suites together. - Technological transitions carry a steep internal self-learning curve; technical staff cannot instantaneously adapt to major AI paradigm shifts. - Cross-functional inclusion is mandatory: leaving operations or end-users out of initial design reviews guarantees that downstream teams will reject the finished tool due to real-world edge cases.

---

[09:45](https://www.youtube.com/watch?v=0xC6tmtyDr8&t=585s) Bridging the Communication Chasm and Respecting Model Assumptions

Communication within an AI initiative is a two-way street that frequently breaks down on both sides. Executive leaders frequently complain that their data scientists and engineers fail to articulate essential project status updates in clear business language. However, Bianco notes that executives rarely evaluate their own communication toward the technical staff. When technical leads conduct multi-hour presentations dense with mathematical theory, business leaders leave baffled; conversely, when executives present high-level business logic that seems crystal clear to them, engineers often walk away struggling to understand why the feature is necessary or how it translates into architectural requirements.

```mermaid flowchart LR accTitle: Two-Way Communication Chasm Between Business and Tech subgraph Business["Business & Executive Leadership"] B1["High-Level Strategy & Commercial Needs"] B2["Frustration with Mathematical Jargon"] end

subgraph Technical["Quant & Engineering Teams"] T1["High-Dimensional Math & Statistical Validation"] T2["Frustration with Vague Functional Specs"] end

Business <-->|"Must Translate Assumptions & Constraints"| Technical ```

Beyond vocabulary mismatches, communication breakdowns around data infrastructure, skill sets, and model assumptions pose severe risks. Bianco draws a direct parallel to the 2007–2008 global financial crisis: quantitative models collapsed not because the math was inherently defective, but because the real-world assumptions baked into those models broke down. Market participants were chasing profits while the lines of communication regarding underlying systemic risks went silent. In enterprise settings today, similar breakdowns occur when leadership overpromises internal capabilities, forcing companies to scramble for expensive emergency contractors when the technical staff hits an architectural wall.

- Communication failures are symmetrical: non-technical managers struggle to convey actionable context, while technical quants overwhelm stakeholders with four-hour mathematical deep dives. - Parallels to the 2007–2008 financial crash: models failed because underlying operational and statistical assumptions were ignored while departments prioritized short-term financial gains over risk transparency. - Unvetted skill assessments cause severe budget overruns when teams overpromise on AI expertise and must later hire expensive outside consultants to salvage execution.

---

[12:11](https://www.youtube.com/watch?v=0xC6tmtyDr8&t=731s) The Three Definitions of a "Deadline" and the Culture of Failure

Organizational culture determines how AI initiatives navigate schedules, setbacks, and technical friction. Bianco shares an experience at an enterprise plagued by conflicting internal semantics, where different departments operated under three distinct definitions of a "deadline": 1. *The Nice-to-Have Date*: A soft target where missing the cutoff (e.g., January 5th) carried no urgency or operational consequence. 2. *The Do-or-Die Commitment*: Bianco’s personal quant camp, which viewed the agreed date as an inviolable, non-negotiable operational necessity. 3. *The Rolling Negotiation Point*: A flexible marker treated merely as an invitation to convene a meeting as the date approached to negotiate a later timeline.

```mermaid flowchart TD accTitle: Three Conflicting Definitions of a Deadline subgraph CultureGap["Cultural Alignment Breakdown"] D1["Definition 1: Nice-to-Have Missed dates carry no consequence"] D2["Definition 2: Do-or-Die Absolute commitment to ship on schedule"] D3["Definition 3: Rolling Negotiation A marker to reschedule as it nears"] end

CultureGap --> Fail["Total Timeline Collapse & Executive Disillusionment"] ```

When disparate corporate divisions collaborate with conflicting understandings of a deadline, overall project failure is preordained; teams are simply floating arbitrary numbers without shared accountability. This cultural dysfunction extends directly to how organizations process technical failure. When leadership assembles an eager team to build an ambitious AI solution, but starves them of adequate infrastructure, clean data, realistic budgets, or properly scoped timelines, the initiative inevitably stalls. If leadership responds by simply declaring the project an embarrassment and blaming the engineering staff, it destroys team morale and sabotages the organization's broader, long-term AI capabilities.

- Conflicting operational definitions of deadlines (soft target vs. hard cutoff vs. renegotiation point) undermine cross-departmental delivery and guarantee project slip. - Punitive responses to project setbacks overlook systemic corporate shortcomings—such as poor data provisioning, insufficient headcount, and unrealistic budgets. - Treating technical failures as individual engineering shortcomings rather than systemic planning deficiencies breeds an organizational culture of risk-aversion and employee burnout.

[13:36](https://www.youtube.com/watch?v=0xC6tmtyDr8&t=816s) Company Culture, Process, and the Friction of Agile

Retaining top talent is impossible if the internal environment is broken. Drawing a parallel to Meta's aggressive recruitment efforts, throwing a roster of industry "all-stars" together onto one team does not guarantee success if there is no unified culture or alignment on shared expectations. When high-caliber professionals come from diverse organizational backgrounds, establishing cohesion is an immense hurdle that leadership must actively address.

Beyond high-level culture, daily processes dictate team morale and productivity. Addressing the standard development workflows used in the software world, Dimitri shares his candid aversion to Agile methodology: "I hate agile. I'm sorry if you're in tech. I can't do it." While acknowledging that some exceptional tech teams thrive under Agile frameworks, he emphasizes that it often clashes with research-driven quant workflows. Equally critical to team sustainability is meaningful recognition: establishing genuine practices for acknowledging hard work, celebrating milestones, and treating contributors with respect.

[14:14](https://www.youtube.com/watch?v=0xC6tmtyDr8&t=854s) Data Reality: Why Volume Doesn't Equal Value and AI Can't Build Without Materials

Without the right data, building effective models is impossible. Dimitri recalls an engagement at a firm where he implemented a straightforward logistic regression to solve an operational challenge. Leadership pushed back, asking, "Why isn't this an AI solution? Why isn't this an XGBoost model?" The reality was simple: the firm lacked both the data volume and data quality required to support a complex machine learning architecture.

To evaluate data readiness, Dimitri relies on the Six Sigma dimensions of data quality: accuracy, completeness, consistency, timeliness, validity, uniqueness, integrity, and reliability. He cautions against the belief that mathematical shortcuts can compensate for missing fundamentals. There is no automated AI tool or kernel density estimation technique capable of generating synthetic data that magically yields a top-tier model. Dimitri compares this misconception to physical construction: "You can't expect a home builder to build a house if they don't have the materials." Furthermore, firms often brag about having 10 million records, 100 million records, or four terabytes of storage, but massive datasets are useless if they contain redundant records and lack unique, predictive signal.

[15:30](https://www.youtube.com/watch?v=0xC6tmtyDr8&t=930s) The End-to-End Modeling Pipeline and Ownership

Effective model implementation requires structuring the entire lifecycle into an interconnected, iterative pipeline. The process must originate with the business units clearly defining the core problem. That specification passes to data engineering to source and prepare inputs, moves to model development teams to engineer the algorithms, proceeds to model validation teams for rigorous risk management testing, and transitions to tech teams for deployment, before looping back to the business for continuous feedback.

```mermaid flowchart LR accTitle: End-to-End Quantitative Modeling Pipeline subgraph S1["Problem & Ingestion"] A["Business Problem Definition"] --> B["Data Engineering"] end subgraph S2["Development & Validation"] B --> C["Model Development"] C --> D["Model Validation & Risk"] end subgraph S3["Production & Loop"] D --> E["Tech & Implementation"] E --> F["Business Feedback Loop"] F -.-> A end ```

Establishing strict, unambiguous ownership across each link in this chain allows organizations to pinpoint operational bottlenecks immediately and resolve friction before projects stall.

[16:09](https://www.youtube.com/watch?v=0xC6tmtyDr8&t=969s) Talent Retention and Making Time for Continuous Learning

Acquiring and retaining elite quantitative and research talent requires more than financial incentives. While large tech firms frequently attempt to lure talent by throwing money at candidates, sustainable retention hinges on intellectual fulfillment. Model developers and quantitative researchers often value stimulating research projects and opportunities to expand their skill sets far more than an extra $5,000 or $10,000 in base compensation.

Fostering an authentic learning environment requires dedicated time, not superficial gestures. Dimitri shares an anecdote about a previous employer that purchased a collection of reference books, placed them on an office bookshelf, and expected the team's skills to improve autonomously:

- Leadership later complained that nobody was reading the books, even accusing Dimitri of ignoring them. - Dimitri opened his backpack to reveal six technical textbooks he was actively carrying around. - He explained that the team was given zero dedicated bandwidth during the workday to study because leadership was constantly rushing them to meet tight deadlines in pursuit of short-term ROI. - Dimitri points out the impracticality of this approach in a fast-moving AI landscape: an engineer who graduated in 2010 cannot be expected to master 2025 generative AI and LLM architectures if the organization allocates no working hours for continuous education.

[17:46](https://www.youtube.com/watch?v=0xC6tmtyDr8&t=1066s) Crawl, Walk, Jog, Run: Making Modeling a Core Competency

Achieving a positive return on AI and ML investments requires disciplined progression: an organization must learn to crawl, then walk, then jog, before it can ever run. Too many enterprises attempt to bypass the fundamentals entirely, aiming to sprint a marathon with minimal internal talent, inadequate data readiness, and an over-reliance on third-party vendor tools.

To avoid falling into the industry benchmark where 70% to 85% of AI/ML projects fail to reach completion, businesses must integrate modeling as a permanent core competency. This requires unifying three key pillars into everyday operations: - **Technical Execution**: Grounding model development in appropriate, high-quality data rather than buzzwords. - **Governance & Risk Management**: Establishing rigorous model validation protocols. - **Business Alignment**: Ensuring business stakeholders properly scope problem definitions and provide active feedback.

## Notable quotes > [00:04](https://www.youtube.com/watch?v=0xC6tmtyDr8&t=4s) Shoot for the target, not for the stars. — Dimitri Bianco

> [01:55](https://www.youtube.com/watch?v=0xC6tmtyDr8&t=115s) build me AI — Dimitri Bianco

> [04:12](https://www.youtube.com/watch?v=0xC6tmtyDr8&t=252s) improve customer experience — Dimitri Bianco

> [06:03](https://www.youtube.com/watch?v=0xC6tmtyDr8&t=363s) die quietly — Dimitri Bianco

> [07:42](https://www.youtube.com/watch?v=0xC6tmtyDr8&t=462s) We do AI. — Vendor Representative

> [13:36](https://www.youtube.com/watch?v=0xC6tmtyDr8&t=816s) I hate agile. I'm sorry if you're in tech. I can't do it. — Dimitri Bianco

> [14:14](https://www.youtube.com/watch?v=0xC6tmtyDr8&t=854s) Why isn't this an AI solution? Why isn't this an XGBoost model? — Corporate Leadership

> [14:14](https://www.youtube.com/watch?v=0xC6tmtyDr8&t=854s) You can't expect a home builder to build a house if they don't have the materials. — Dimitri Bianco

## Mentioned Dimitri Bianco, Fancy Quant, Ai4, Wall Street, ChatGPT, Meta, XGBoost, Six Sigma

## Video description Dimitri Bianco, Quant Founder, Fancy Quant

Abstracts

Are you spending too much money on AI/ML initiatives only to fall short on generating meaningful solutions? Generating positive ROI on AI/ML is all about building core competencies and getting small wins. Many firms are chasing the AI hype without clear goals or paths to get meaningful results. I will provide insights on getting small wins and how to build AI as a core part of the business instead of a special project.

[Source](https://www.youtube.com/watch?v=0xC6tmtyDr8)

中文简报

Bianco警告：目标不明致85%的AI项目ROI落空企业AI项目的溃败主要源于未明确的成功指标、权责分散和数据质量不佳，而非数学建模缺陷。 Dimitri Bianco深入剖析了为何70%至85%的企业AI及机器学习项目未能实现正向ROI，并将这些失败归因于规划缺陷、沟通脱节、数据质量不足以及技能不匹配。他警示企业切勿落入耗时多年的整体式自动化陷阱，亦不要轻信供应商的空洞承诺，而是敦促管理层聚焦于高杠杆、易见效的小型突破，例如利用LLM编写监管模型文档。要想取得成功，组织必须在整个建模流程中落实单一职责机制，重视数据质量的基础维度，并投资于员工的持续技能提升。归根结底，把AI转化为可持续的业务驱动力，需要遵循从‘爬、走、慢跑到奔跑’的严谨递进路径，并以技术严谨性、验证治理及明确的业务契合度为坚实后盾。 AI 与机器学习项目的失败率高达 70% 至 85%，其核心根源通常并非技术上未能交付，而是糟糕的投资回报率（ROI）、严重的预算超支以及交付延期。 (0:04) 企业切忌盲目跟风追求耗时数年的全流程模型自动化，而应将工作流拆解为高杠杆的小颗粒度切入点（例如利用大模型自动化生成合规文档）以实现切实回报。 (4:12) 由于缺乏明确的单点责任人（single-threaded ownership），大量企业级 AI 项目在跨部门沟通壁垒和优先级漂移中陷入停滞并“悄无声息地消亡”。 (6:03) 单纯庞大的数据规模并不能直接带来业务价值，缺乏准确度与独特性的数据无法通过合成数据或数学捷径掩盖，企业必须严守数据质量基本功。 (14:14) 企业无法依靠拼凑外包工具一步登天，而必须遵循“爬行、行走、慢跑、奔跑”的渐进路径，将技术实施、风控验证与业务对齐深度融合为组织的长期核心竞争力。 (17:46)

Reading summary

Dimitri Bianco argues that enterprise AI initiatives suffer 70% to 85% failure rates largely due to misaligned planning, broken communication, and poor data readiness rather than deficient mathematics. Instead of pursuing monolithic, hype-driven automation moonshots, organizations must treat quantitative modeling as an incremental, disciplined capability focused on targeted operational bottlenecks. Economic Realities Behind AI Project Failures (0:04) Bianco observes that while standard enterprise technology projects fail at rates between 25% and 50%, artificial intelligence and machine learning initiatives experience failure rates of 70% to 85%. Crucially, failure is typically an economic collapse of return on investment (ROI)—such as spending $2 million to finish a $500,000 project—rather than outright abandonment. These breakdowns trace back to four root causes: planning, communication, data, and skills. A primary driver is executive hype: leadership demands AI without defining measurable business problems or explicit success metrics. Organizations also consistently misjudge the resource triad of time, people, and technology. Bianco stresses that project timelines are almost universally underestimated because teams overlook hidden data hygiene issues, infrastructure obstacles, and the steep internal learning curves required to operationalize new paradigms. Targeted Wins Over Monolithic Moonshots (4:12) Bianco cautions against grand, multi-year automation initiatives. He cites a major financial institution that spent five to seven years and fortunes of capital attempting to fully automate the quantitative modeling lifecycle. The project was abruptly canceled by an incoming CEO because leadership treated quantitative research as a simple, linear process rather than an intricate, iterative discipline of hypothesis testing and statistical validation. In contrast, sustainable ROI comes from decomposing complex workflows into bite-sized friction points. In regulated banking, quantitative teams routinely write 1,000 to 2,000 lines of code but face the burden of producing 70- to 80-page compliance dossiers. Deploying large language models (LLMs) to convert technical code annotations into standardized reporting templates automates a massive administrative bottleneck while preserving human review, creating tangible value where broad slogans like "improve customer experience" fail. Communication Chasms and Conflicting Deadlines (9:45) AI projects frequently suffer from a symmetrical communication breakdown. While executives struggle with four-hour technical presentations laden with mathematical theory, quantitative teams are left bewildered by high-level business concepts that lack clear functional specifications. Drawing a parallel to the 2007–2008 financial crisis, Bianco warns that models collapse when underlying operational and statistical assumptions are ignored in the pursuit of short-term commercial returns. These divides are aggravated by conflicting internal definitions of accountability. Bianco notes corporate teams often operate with three incompatible views of a "deadline": a soft "nice-to-have" date, a strict "do-or-die" cutoff, and a "rolling negotiation point." Without single-threaded ownership, projects experience "quiet deaths": secondary data teams quietly drop months-long extraction tickets as priorities shift, leaving models shelved in limbo until executive inquiries trigger cross-departmental blame games. Data Realities, Tooling Traps, and Maturity (14:14) A core misconception in enterprise AI is equating data volume with predictive value. Organizations often tout terabytes of storage or millions of records, but datasets are useless if they lack unique signal or fail the Six Sigma data quality dimensions: accuracy, completeness, consistency, timeliness, validity, uniqueness, integrity, and reliability. Bianco emphasizes that synthetic data cannot magically replace missing fundamentals, comparing the practice to asking a builder to construct a house without materials. Where data is limited, simple methods like logistic regression are far more appropriate than forced machine learning models like XGBoost. Furthermore, Bianco warns against vendor platforms that masquerade expensive consulting engagements as turnkey software, which frequently saddle firms with unmaintainable integration stacks. Organizations must adopt a disciplined "crawl, walk, jog, run" progression, integrating business problem definition, data engineering, model development, validation, and deployment into an end-to-end feedback loop while granting technical talent dedicated workday hours for continuous education. Bianco concludes that achieving sustained ROI in AI requires abandoning vendor hype and monolithic bets in favor of strong data fundamentals, rigorous governance, and cross-functional ownership. Organizations cannot sprint into advanced modeling without first mastering the organizational crawl and walk phases.

精炼阅读摘要

量化金融从业者Dimitri Bianco指出，企业AI/ML项目面临70%至85%的惊人失败率，其核心原因往往不是数学或技术缺陷，而是规划脱节、权责不清、数据质量不足及双向沟通断层。企业必须放弃对LLM等热门概念的盲目追逐，聚焦可度量、小步快跑的业务痛点，将端到端建模能力内化为组织的核心竞争力。高失败率根源与投资回报陷阱 (0:04) 传统企业IT项目的失败率通常在25%到50%之间，而AI和机器学习项目的失败率则高达70%到85%。Bianco指出，AI项目的“失败”往往并非指系统未能上线，而是严重的投资回报率（ROI）灾难。例如，一个原定预算50万美元的开发管线，最终却耗费了200万美元才勉强交付，在商业层面上即是彻底的失败。这类问题的根源始终可以归结为四个基本支柱的坍塌：规划、沟通、数据与人才技能。许多管理层受到行业展会与生成式AI炒作的影响，在缺乏清晰商业定义和可量化指标的前提下盲目要求团队“构建AI”，最终导致即便模型成功部署，业务部门也因缺乏前期设定的成功标准而将其判定为无效投资。拆解宏大构想：警惕自动化陷阱 (4:12) 宏大而缺乏实操性的全盘自动化构想往往会吞噬巨额资源。Bianco以一家大型金融机构为例：该机构在生成式AI爆发前曾启动一项耗资巨大、历时5至7年的浩大工程，试图将量化模型开发全生命周期彻底自动化并消灭人工环节，最终新任CEO因项目耗费巨资却见不到实际收益而愤怒叫停。这一失败源于管理层忽视了量化建模本质上是包含假设检验、高维统计探索与持续验证的复杂迭代过程，而非单一步骤的机械流水线。与其追求“改善客户体验”这类空洞口号，企业应将问题拆解为高价值的小切口。例如大型银行的量化工程师虽然编写1000至2000行复杂代码，但常被长达70至80页的监管合规文档所累；利用LLM将工程师的日常草稿笔记自动整理为符合合规模板的标准文档，便能以极高ROI消除沉重的运营瓶颈。权责稀释与项目悄无声息的消亡 (6:03) 许多企业AI项目并未伴随轰轰烈烈的警报宣告破产，而是在组织壁垒中“悄无声息地死去”。典型场景是业务领导发起AI项目后，数据部门评估提取数据表需要3至4个月；随后企业战略重点转移，数据团队搁置需求，量化团队在漫长等待中将模型开发边缘化，数月后董事会询问进展时，管理层才惊觉项目已无疾而终。这一过程伴随着部门间的相互推诿：数据工程指责IT基础设施，量化人员抱怨输入数据劣质，业务端则指责技术交付滞后。此外，组织内部对“截止日期”理解的割裂进一步加剧了执行崩溃——部分部门视其为不具约束力的软目标，量化团队视其为必须死守的底线，而另一些团队则将其当作重新谈判延期的节点。若缺乏贯穿全流程的单一负责人（single-threaded owner），跨部门协作必然陷入停滞。跨部门沟通鸿沟与模型底层假设 (9:45) AI项目中的沟通障碍是双向对等的。业务管理者常抱怨数据科学家只会进行充斥着高维数学理论的数小时汇报，无法用商业语言同步进展；反过来，管理者自以为清晰的宏观业务构想，在工程师眼中却往往是一团无法落地为系统架构规范的模糊逻辑。Bianco将这种脱节与2007至2008年全球金融危机进行类比：当时量化模型的崩溃并非由于数学公式本身的算力缺陷，而是因为模型所依赖的现实市场假设已彻底失效，而追逐短期利润的机构却切断了对系统性风险的透明沟通。在企业落地中，如果忽视模型基础假设或夸大内部技术实力，往往会导致后期不得不雇佣极其昂贵的外部顾问来仓促救场。数据质量现实与全流程治理管线 (14:14) 没有合格的数据原料，再高级的算法也无从谈起。Bianco强调，企业常炫耀自身拥有数百万条记录或数TB存储，但如果缺乏独特且具备预测性的信号，海量数据依然毫无价值。试图通过核密度估计等数学捷径生成合成数据来弥补基础数据的缺失，如同指望建筑工人在没有建材的情况下凭空盖楼。构建可持续的AI能力必须依赖严格界定权责的端到端管线：从业务部门定义核心问题出发，依次推进至数据工程清洗、模型开发构建、模型验证部门进行严苛的风险测试、技术团队系统部署，最终再形成业务反馈闭环。与此同时，企业必须摆脱对第三方供应商“全包式AI”噱头的过度依赖，摒弃照搬敏捷开发带来的研发冲突，并在工作时间内为员工提供充足的技术学习带宽，遵循“爬、走、慢跑、快跑”的规律，稳步积累组织自身的工程与治理能力。要打破70%到85%的AI项目失败魔咒，企业必须停止追逐华而不实的新奇技术，转向稳扎稳打的能力建设。这要求组织将技术执行（立足于高质量数据而非噱头）、风险治理（严谨的模型验证）以及业务对齐（精准定义问题并敏捷反馈）三者有机融为一体，真正把AI建模转化为自身长期的核心竞争力。

Detailed notes — zh

## Background 该内容由行业峰会频道 Ai4 于 2025 年 8 月 26 日发布在 YouTube 平台上，演讲者为 Fancy Quant 创始人 Dimitri Bianco。视频简介指出，当前许多企业在 AI/ML 计划中过度投入却收效甚微，演讲旨在分享如何通过积累小胜、培养核心能力，将 AI 真正内化为企业核心业务而非独立的特殊项目，从而实现正向的投资回报率（ROI）。 ## TL;DR Fancy Quant 创始人 Dimitri Bianco 剖析了 AI/ML 项目失败率高达 70% 至 85% 的深层原因，指出行业普遍陷入过度迷信大语言模型（LLM）与好高骛远的误区。他从规划不周、沟通断层、基础设施与数据脱节、团队技能不足四大维度切入，强调高质量数据胜过盲目追求复杂算法，并倡导化整为零的渐进式落地策略。演讲最后呼吁企业建立端到端的模型研发流水线与包容的学习型文化，遵循“先爬后跑”的规律，将模型治理与风险管理转化为内部核心能力以获取切实回报。 ## Atmosphere & style 演讲整体呈现出一种务实、辛辣且直击痛点的行业老手姿态，充满对当前 AI 狂热泡沫的清醒审视。Dimitri 结合自身在华尔街与大型银行的一线量化风险经验，抛出诸如“截止日期的三种定义”、供应商“宣称会做 AI 实则是昂贵咨询”等荒诞轶事，叙事生动风趣却直戳企业管理软肋。整场分享节奏紧凑、直截了当，不仅揭露了跨部门推诿与项目悄然夭折的残酷现实，更带有一种倾囊相授的真诚警醒感。 ## Key takeaways - [00:04](https://www.youtube.com/watch?v=0xC6tmtyDr8&t=4s) AI/ML 项目的失败率高达 70% 至 85%，远高于传统 IT 项目的 25% 至 50%，核心问题在于投资回报率（ROI）严重崩盘和预算失控。 - [00:04](https://www.youtube.com/watch?v=0xC6tmtyDr8&t=4s) 导致 AI 项目折戟的四大核心根源是：规划不周、沟通断层、数据与基础设施脱节以及团队技能不足。 - [02:17](https://www.youtube.com/watch?v=0xC6tmtyDr8&t=137s) 切忌好高骛远设定全自动化等宏大目标，应当化整为零（Bite-sized pieces），从小切口（如监管文档自动化生成）快速获取小胜与正向 ROI。 - [06:03](https://www.youtube.com/watch?v=0xC6tmtyDr8&t=363s) 缺乏明确的项目所有权（Ownership）会导致部门间互相推诿，甚至让关键项目因排期和优先级变动而“悄无声息地流产”。 - [07:42](https://www.youtube.com/watch?v=0xC6tmtyDr8&t=462s) 警惕外部供应商兜售的万能 AI 工具，过度拼凑系统容易陷入兼容噩梦，其实质往往是昂贵的咨询项目而非即插即用的解决方案。 - [07:42](https://www.youtube.com/watch?v=0xC6tmtyDr8&t=462s) 技术与业务团队必须建立双向对齐机制，避免技术人员沉迷术语细节而业务人员无法阐明客户实际价值的沟通断层。 - [10:42](https://www.youtube.com/watch?v=0xC6tmtyDr8&t=642s) 企业内部若对“截止日期（Deadline）”缺乏统一严肃的共识，或者在遭遇挫折时粗暴归咎于技术能力，将彻底扼杀创新土壤。 - [14:14](https://www.youtube.com/watch?v=0xC6tmtyDr8&t=854s) 数据质量与独特性远重于数据量，低质数据下盲目上马复杂算法（如 XGBoost）毫无意义，务实的简单模型（如逻辑回归）往往更有效。 - [15:30](https://www.youtube.com/watch?v=0xC6tmtyDr8&t=930s) 应将机器学习研发打造成覆盖业务定义、数据工程、模型开发、独立验证、系统部署与持续反馈的端到端流水线并明确各环节权责。 - [16:09](https://www.youtube.com/watch?v=0xC6tmtyDr8&t=969s) 保留顶级量化与算法人才不能只靠薪酬，更要提供探索性研究文化并在工时中预留充足的学习时间以跟上前沿技术演进。 - [17:46](https://www.youtube.com/watch?v=0xC6tmtyDr8&t=1066s) 企业在 AI 领域必须遵循从爬、走到跑的演进规律，唯有将建模、治理与风险管理内化为核心能力，才能真正规避失败并实现长期正向 ROI。 ## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=0xC6tmtyDr8&t=4s) 量化背景与 AI 项目的高失败率现状 Fancy Quant 创始人 Dimitri Bianco 在开场中首先明确了自己的专业背景——量化金融（Quantitative Finance）。在华尔街和好莱坞电影的叙事里，量化人员（Quant）通常是那些负责构建数学、统计学和复杂模型的群体；但在当下的金融实践中，量化、机器学习与 AI 工程早已高度融合。Bianco 指出，整场大会似乎都在言必称大语言模型（LLM），但他呼吁大家退后一步审视全局：AI 的范畴远比 LLM 广阔得多，行业正在忽视大量其他实用工具。这就好比是“拿着方榫死命往圆孔里锤”，试图用 LLM 去生搬硬套所有业务场景，这正是本次演讲主题“瞄准靶心，而非好高骛远（Shoot for the Target, Not the Stars）”的由头。 ```mermaid flowchart TD accTitle: 技术与AI项目失败率对比 subgraph S1["传统 IT 技术项目"] A1["标准项目交付"] --> B1["25% - 50% 失败率"] end subgraph S2["AI / ML 创新项目"] A2["AI/ML 项目交付"] --> B2["70% - 85% 失败率"] B2 --> C2["投资回报率 (ROI) 崩塌 （如预算50万实耗200万）"] end ``` 根据 Bianco 历任量化风险与研究负责人、在各大跨国银行及国家级银行工作的经验，标准 IT 技术项目的失败率通常在 25% 到 50% 之间；而 AI 项目的失败率则高达 70% 至 85%。他强调，这里的“失败”往往并非指项目彻底无法上线，更核心的痛点在于投资回报率（ROI）的严重崩盘——例如原本规划耗资 50 万美元并设有明确截止日期的项目，最终却花费了 200 万美元才勉强做完，“即便把东西硬做出来了，这也是极其糟糕的 ROI”。导致 AI 项目折戟沉沙的核心原因可以归结为四大维度：规划（Planning）、沟通（Communication）、数据与基础设施（Data/Infrastructure）以及技能（Skills）。 - **传统 IT 项目失败率**：25% – 50% - **AI/ML 项目失败率**：70% – 85% - **核心失败根源**：规划不周、沟通断层、数据与基础设施脱节、团队技能不足 --- ### [02:17](https://www.youtube.com/watch?v=0xC6tmtyDr8&t=137s) 规划不足：目标过载与“一口吃成胖子”的陷阱在谈及规划（Planning）时，Bianco 直言许多企业高管在参加完技术峰会后便热血沸腾，回到公司直接提出“Dimitri，快来给我们做 AI”这样空洞的要求，却根本没有想过具体的业务用例、要解决的痛点规模以及衡量成功的量化指标（Success Metrics）。很多时候，量化开发团队兴高采烈地上线了自认为达标的模型，业务部门却怨声载道，抱怨完全不符合实际工作流。这种脱节的根源在于项目启动之初就未界定成功的标准，导致双方预期完全错位；加之缺乏明确的项目所有权（Ownership）划分，以及在时间、人员与技术资源上的严重误判（尤其是时间预估，AI 项目中几乎百分之百会出现时间延期）。 ```mermaid flowchart LR accTitle: 目标设定策略对比 subgraph BAD["高风险：宏大且模糊的目标"] A["试图完全自动化 整个模型开发流程"] --> B["耗时 5-7 年 / 耗资巨大"] B --> C["忽视复杂性 以 CEO 震怒失败告终"] end subgraph GOOD["高可行性：切碎为小目标 (Bite-sized)"] D["聚焦模型合规文档撰写"] --> E["利用 LLM 整理笔记 套用 70-80 页监管模板"] E --> F["人工最终校验 小步快跑实现 ROI"] end ``` Bianco 分享了一个亲身经历的反面教材：在 LLM 爆发前，某家大型金融巨头曾立项试图“全自动完成量化模型开发”，试图完全取代量化团队的工作。该项目耗时 5 到 7 年，耗费了巨额资金，最终新任 CEO 介入审查时大发雷霆，因为整个团队投入巨大却一无所获。失败的症结在于项目负责人把极其复杂的量化模型开发误以为是“单一步骤”，全然不知其包含成百上千行代码、复杂统计测试、维度分析与数学空间推演等繁复环节。相比之下，Bianco 目前在大型银行的朋友采取了“化整为零（Bite-sized pieces）”的策略：银行业监管极其严苛，一份模型合规文档往往长达 70 至 80 页；量化工程师在边写代码边做笔记后，利用 LLM 将零散记录自动整理并填充进标准文档模板中，最后由人工通读审核。这种聚焦于文档自动化的小切口，远比高喊“我们要提升客户体验”这种空中楼阁式的口号更具实操价值。 - **全自动化模型开发教训**：耗资巨大且耗时 5-7 年，最终因目标过于脱离现实而惨淡收场 - **监管文档规模**：大型受监管银行的模型说明文档通常长达 70–80 页 - **有效切入点**：利用 LLM 辅助整理笔记与格式化文档，而非幻想一步到位取代核心开发 --- ### [06:03](https://www.youtube.com/watch?v=0xC6tmtyDr8&t=363s) 责任归属缺位与项目“悄然流产” 缺乏明确的责任人（Ownership）是导致 AI 项目中途瓦解的另一大致命伤。在很多企业中，一旦开发遇到阻碍，各部门便会陷入互相甩锅的泥潭：“为什么数据部门给的数据这么差？”“为什么 IT 基础设施没有跟上？”“为什么模型开发进度落后这么多？”这种内耗往往并非源于不可克服的技术难题，而是因为项目在规划阶段就没有清晰界定每个环节的最终负责人，最终导致无休止的低效会议、员工职业倦怠（Burnout）以及前期投资的大量浪费。更荒谬的现象是项目的“悄无声息地死掉（Projects die quietly）”。前线业务或技术发起人起初兴致勃勃推进项目，但当他们向数据部门提出需求时，被告知需要排期 3 到 4 个月；在此期间，公司内部优先级发生变动，数据团队将该需求搁置，量化与机器学习团队因为久等无果也顺势调低了其优先级。几个月后，董事会或高管会议上突然有人问起该项目的进展，所有人才尴尬地发现项目早已名存实亡，负责人只能现场编造借口试图重新启动。Bianco 指出，建立清晰的项目归属机制并维持严密的项目管理追踪，是防止项目在部门推诿中无疾而终的关键。 ```mermaid flowchart TD accTitle: 项目悄然流产的过程 A["项目发起人兴致勃勃立项"] --> B["向数据部门索取数据"] B --> C["数据部门回复需排期 3-4 个月"] C --> D["公司优先级调整 / 数据需求被搁置"] D --> E["算法团队因等待过久降低优先级"] E --> F["董事会问责项目进展"] F --> G["项目已在沉默中实质性死亡"] ``` --- ### [07:42](https://www.youtube.com/watch?v=0xC6tmtyDr8&t=462s) 供应商盲信与跨部门沟通壁垒在资源调配方面，Bianco 告诫企业千万不要过分迷信外部供应商的工具产品。他在展会现场亲身经历了一次荒谬对话：一家宣称能解决所有业务难题的供应商找上门来，当 Bianco 询问对方具体做什么时，对方答复称“我们做 AI（We do AI）”；进一步追问后，对方表示可以派人入驻企业观察数据、诊断业务问题并开发方案。Bianco 调侃道：“这听起来完全就是极其昂贵的咨询项目，而不是什么即插即用的 AI 工具。”企业如果试图将市面上所谓“最好的”数据系统、基础设施和项目工具拼凑在一起，往往会陷入系统互不兼容的噩梦。此外，技术的更新迭代要求团队保持自主学习，例如像 ChatGPT 这样的大模型版本升级，业务层不能指望技术团队在一夜之间无缝掌握全新范式。 ```mermaid flowchart LR accTitle: 跨部门沟通断层 subgraph TECH["技术团队 (Quant / ML)"] T1["沉迷底层细节 滔滔不绝讲4小时技术术语"] end subgraph BIZ["业务团队 (高管 / COO / 运营)"] B1["提出抽象需求 无法阐明功能真正的业务价值"] end T1 <-->|沟通断层：缺乏双向提问与前置对齐| B1 ``` 沟通障碍更是贯穿技术与业务之间的双向断层。一方面，高管和 COO 常抱怨技术人员讲不清楚关键信息；另一方面，技术人员在做汇报时往往一讲就是 4 个小时复杂的数学统计细节，台下的业务人员则听得一头雾水。同样地，业务端有时自认为需求表达得极其清晰，但在开发人员听来却完全无法理解该功能对客户的具体价值。如果项目只把量化与算法团队关在房间里闭门造车，而忽略了运营（Operations）等关键利益相关方，最终做出的方案很可能会在交付给运营团队时被列出 10 条无法落地的理由而彻底作废。因此，在资源筹备期必须让核心部门代表共同在场，进行充分的双向沟通与提问。 --- ### [10:42](https://www.youtube.com/watch?v=0xC6tmtyDr8&t=642s) 基础设施、技能高估与“截止日期”的文化冲突跨部门协作的常见阻碍还包括数据基础设施的短板、模型底层假设的失效以及技能的盲目高估。许多企业自以为部署了 AWS 或云端环境就万事大吉，实则根本不具备运行高阶模型所需的底层架构，导致中途追加采购、延误数周乃至数月。Bianco 提到了 2007-2008 年全球金融危机的惨痛教训：当时华尔街众多量化模型全面崩溃，各方互相推卸责任，其深层原因正是模型内包含的大量假设在极端行情下失效；虽然许多人早就知晓这些假设的局限，但在追逐利润的狂热中，跨部门之间关于风险假设的沟通机制彻底缺位。此外，部分团队在立项时高估了自身技能，中途不得不紧急雇佣昂贵的外部顾问补救，使得原本的预算和时间彻底失控。 ```mermaid flowchart TD accTitle: 同一企业内部关于截止日期的三种定义冲突 D["公司内部 Deadline 概念冲突"] D --> D1["【定义 1：最好能达成的参考日期】 1月5日能做完最好，不行就延后"] D --> D2["【定义 2：誓死捍卫的硬性底线】 1月5日必须交付，绝无妥协余地"] D --> D3["【定义 3：动态谈判的起步价】 先定1月5日，临近日期再重新协商"] ``` 在企业文化层面，Bianco 举了一个令他极其恼火的真实案例：某家公司内部对“截止日期（Deadline）”竟然存在三种截然不同的理解： 1. **参考型（Nice to have）**：认为 1 月 5 日只是个理想目标，做不完就往后推，无伤大雅； 2. **死磕型（Do or die）**：以 Bianco 为代表的专业阵营，认为 1 月 5 日是不可逾越的铁律，必须按期交付； 3. **谈判型（Negotiable）**：认为 1 月 5 日只是一个初步时间节点，随着日期临近随时可以重新谈判延期。当不同部门、项目成员乃至高管在缺乏统一认知的情况下共事，项目失败便成了必然结局，因为大家只是在纸面上写下一个漂亮的数字，暗中指望别人去实现罢了。Bianco 最后强调，企业在面对初创团队或初涉 AI/ML 领域的员工时，如何对待失败（How failures are treated）将直接决定团队的生死存亡。如果一旦因为缺钱、缺数据或缺资源导致项目受挫，管理层就粗暴地将其定性为“项目很烂、开发团队能力不行”，这种文化将彻底摧毁技术创新的土壤。 ### [13:36](https://www.youtube.com/watch?v=0xC6tmtyDr8&t=816s) 团队文化、敏捷管理的局限与正向激励即便企业把一群全明星级别的顶尖人才招募进同一个团队，如果缺乏明确的预期与凝聚力强大的团队文化，这些员工根本不会长久留下来。正如当前 Meta（Facebook）所面临的人才管理挑战一样，在不同企业背景的成员之间建立统一的共识本身就是巨大挑战。在日常项目管理中，敏捷开发（Agile）虽然在很多优秀的技术团队中被奉为圭臬且运转良好，但它并不适合所有人和所有业务场景。与其盲目套用流程，不如关注更核心的团队管理要素：如何正向反馈、认可员工的工作成果并共同庆祝阶段性的胜利。 - **全明星团队的文化挑战**：单靠高薪拼凑顶级人才无法自动形成战斗力，建立统一的文化与清晰的预期才是留住人才的关键。 - **对敏捷模式的反思**：敏捷流程并非灵丹妙药，部分团队在生搬硬套敏捷机制时反而会降低研发效能。 - **认可与庆祝（Acknowledgement & Celebrations）**：真诚肯定员工的贡献并给予成就感，是维持团队士气的重要支柱。 --- ### [14:14](https://www.youtube.com/watch?v=0xC6tmtyDr8&t=854s) 数据质量胜于数量：没有好材料就建不成好房子在 AI 和机器学习项目中，数据直接决定了模型的上限——“如果你没有合适的数据，就绝不可能构建出合适的模型”。很多企业盲目追求前沿复杂的算法，比如在明明数据不足、质量堪忧的情况下，质问为什么不用 XGBoost 或复杂的 AI 架构，而 Dimitri Bianco 曾通过构建一个极简的逻辑回归（Logistic Regression）便解决了困扰对方的“神奇业务难题”。单纯宣称拥有“1000 万条记录、1 亿条记录或 4TB 数据”毫无意义，若数据内部缺乏独特有效的信息，庞大的数据量充其量只是冗余。没有任何一种 AI 机制或核密度估计（Kernel Density Estimation）等合成数据手段能够凭空创造出世界级模型。这就好比建筑工人手头缺乏合格的砖瓦建材，自然无法指望他们盖出稳固的房屋。 ```mermaid flowchart TD accTitle: 数据质量与模型产出的依赖关系 subgraph DataQuality["六西格玛数据质量维度"] D1["准确性 Accuracy / 完整性 Completeness"] D2["一致性 Consistency / 及时性 Timely"] D3["有效性 Validity / 唯一性 Uniqueness"] D4["完整性 Integrity / 可靠性 Reliability"] end subgraph RealityCheck["数据现实与模型构建"] Raw["原始数据 (盲目追求 4TB / 1亿条记录)"] --> Filter{"是否具备高质量与独特性?"} Filter --"否 (缺乏信息)"--> Fail["复杂 AI/XGBoost 无法发挥效用"] Filter --"是 (高质量建材)"--> Model["适配模型 / 成功解决业务问题"] end ``` - **六西格玛数据质量标准**：评估数据需考量准确性（Accuracy）、完整性（Completeness）、一致性（Consistency）、及时性（Timely）、有效性（Validity）、唯一性（Uniqueness）、真实完整性（Integrity）与可靠性（Reliability）。 - **破除复杂模型迷信**：面对低质或稀疏数据，复杂的深度学习或集成算法往往适得其反，务实的简单统计模型反而是更优解。 --- ### [15:30](https://www.youtube.com/watch?v=0xC6tmtyDr8&t=930s) 端到端的模型研发流水线与职责划分机器学习研发应当被视为一条严密协作的端到端流水线（Pipeline），而不是孤立的技术实验。首先由业务部门在前端明确定义问题，然后交由数据工程团队处理并确保数据就绪；紧接着由模型开发团队进行算法构建，再经由独立的模型验证团队结合风险管理规范进行严谨的合规与准确性校验；随后交付技术工程团队上线部署；最后由业务端收集真实反馈并反哺迭代。 ```mermaid flowchart LR accTitle: 端到端模型研发流水线架构 subgraph ProblemData["业务定义与数据准备"] BizIn["业务端 (定义业务问题)"] --> DataEng["数据工程 (数据清洗与供给)"] end subgraph ModelingRisk["开发与风险管理"] DataEng --> ModelDev["模型开发团队 (构建算法模型)"] ModelDev --> ModelVal["模型验证团队 (风险管理与合规校验)"] end subgraph DeployFeedback["上线与持续反馈"] ModelVal --> TechImp["技术与部署 (系统工程化落地)"] TechImp --> BizOut["业务端反馈 (迭代优化)"] BizOut -.-> BizIn end ``` - **全流程所有权（Ownership）**：流水线上的每个环节都必须有明确的负责人，只有责任界定清晰，才能快速定位研发瓶颈并高效排除故障。 - **模型验证与风险控制**：独立的模型验证（Model Validation）机制是金融量化与严肃 AI 应用中不可或缺的防线。 --- ### [16:09](https://www.youtube.com/watch?v=0xC6tmtyDr8&t=969s) 人才保留的秘诀：研究文化与学习时间的重要性在模型开发和量化研究领域，顶尖人才的留存往往不能单纯依赖金钱砸出来的薪酬。对许多核心研发人员而言，能够参与一个充满趣味且富有挑战的研究项目，其吸引力远大于“多给 5,000 到 10,000 美元的薪酬”。打造一个包容探索、拥抱学习的团队环境，才是让人才常年追随的核心原因。很多企业在支持员工学习上流于表面。Dimitri Bianco 分享了一个生动的前司轶事：公司买了一大堆专业书籍堆在书架上，对员工说“祝你们好运”，但根本无人翻阅；管理层甚至当面质问 Dimitri 为什么不看书，Dimitri 打开自己的背包，里面随身装着六本厚重的教科书。他直言不讳地指出：“大家不看这些书，是因为我们根本没有时间！”在紧迫的项目截止日期和最大化资产回报率（ROA）的重压下，员工连喘息的机会都没有。在技术以惊人速度演进的今天，如果不给员工预留学习时间，企业无异于在苛求员工凭空掌握一切：“试想一下，你招募了一个 2010 年毕业的员工，指望他在 2025 年掌握大语言模型（LLM）和生成式 AI（GenAI）的所有前沿技术，却完全不给他任何学习时间，这根本不切实际。” - **非物质激励与研究氛围**：给予研发人员探索前沿技术和攻克高难度课题的自由度，是抵御人才流失的强效护城河。 - **虚假的学习支持 vs. 真正的资源赋能**：仅仅提供书架和学习资料是形式主义，合理调配工时、给员工留出消化前沿知识的“学习预算”才是关键。 --- ### [17:46](https://www.youtube.com/watch?v=0xC6tmtyDr8&t=1066s) 从爬到跑：将建模内化为企业核心能力以获取正向 ROI AI 与量化项目的成功必须遵循渐进规律：先学会爬，再学会走，接着慢跑，最后才能全速冲刺（"Learn to crawl, walk, jog, and then run"）。当前业界最大的误区在于：大量企业连“爬”都不会，既没有系统的技能储备，人才团队也严重匮乏，却盲目狂踩油门想要跑马拉松，寄希望于买一堆高阶工具来弯道超车并击败竞争对手——这种做法注定会失败。企业要想摆脱“70% 到 85% 的 AI 项目最终烂尾流产”的行业魔咒，唯一路径是将建模工程转化为企业内部扎实的核心能力（Core Competency）。这不仅涉及技术层面的模型构建，更涵盖严格的模型治理（Governance）、健全的风险管理框架以及业务端精准定义问题的能力。只有把这些基础地基打牢，企业才能在 AI 投资中真正实现正向的投资回报率（Positive ROI）。 - **循序渐进的成熟度曲线**：跳过基础积累而直接追求高精尖方案，是项目失败的根源。 - **核心能力四要素**： 1. **技术研发能力**：算法构建与工程化实现。 2. **业务问题定义**：精准锚定业务痛点，设定合理预期。 3. **模型治理体系**：规范开发流程与资产管理。 4. **风险管理实践**：保障模型的合规性、稳健性与安全性。 - **规避 70%~85% 的失败率**：只有将建模全流程融入企业血脉，才能持续从小胜走向大胜，获得切实的商业回报。 ## Notable quotes > [00:04](https://www.youtube.com/watch?v=0xC6tmtyDr8&t=4s) 拿着方榫死命往圆孔里锤 — Dimitri Bianco > [00:04](https://www.youtube.com/watch?v=0xC6tmtyDr8&t=4s) 瞄准靶心，而非好高骛远（Shoot for the Target, Not the Stars） — Dimitri Bianco > [00:04](https://www.youtube.com/watch?v=0xC6tmtyDr8&t=4s) 即便把东西硬做出来了，这也是极其糟糕的 ROI — Dimitri Bianco > [02:17](https://www.youtube.com/watch?v=0xC6tmtyDr8&t=137s) Dimitri，快来给我们做 AI — Dimitri Bianco > [06:03](https://www.youtube.com/watch?v=0xC6tmtyDr8&t=363s) 悄无声息地死掉（Projects die quietly） — Dimitri Bianco > [07:42](https://www.youtube.com/watch?v=0xC6tmtyDr8&t=462s) 我们做 AI（We do AI） — 外部供应商 > [14:14](https://www.youtube.com/watch?v=0xC6tmtyDr8&t=854s) 如果你没有合适的数据，就绝不可能构建出合适的模型 — Dimitri Bianco > [16:09](https://www.youtube.com/watch?v=0xC6tmtyDr8&t=969s) 大家不看这些书，是因为我们根本没有时间！ — Dimitri Bianco > [16:09](https://www.youtube.com/watch?v=0xC6tmtyDr8&t=969s) 试想一下，你招募了一个 2010 年毕业的员工，指望他在 2025 年掌握大语言模型（LLM）和生成式 AI（GenAI）的所有前沿技术，却完全不给他任何学习时间，这根本不切实际。 — Dimitri Bianco > [17:46](https://www.youtube.com/watch?v=0xC6tmtyDr8&t=1066s) 先学会爬，再学会走，接着慢跑，最后才能全速冲刺（Learn to crawl, walk, jog, and then run） — Dimitri Bianco ## Mentioned Dimitri Bianco, Fancy Quant, Ai4, LLM, ChatGPT, Meta, Facebook, AWS, XGBoost, Logistic Regression, Kernel Density Estimation, Six Sigma ## Video description Dimitri Bianco, Quant Founder, Fancy Quant Abstracts Are you spending too much money on AI/ML initiatives only to fall short on generating meaningful solutions? Generating positive ROI on AI/ML is all about building core competencies and getting small wins. Many firms are chasing the AI hype without clear goals or paths to get meaningful results. I will provide insights on getting small wins and how to build AI as a core part of the business instead of a special project. [Source](https://www.youtube.com/watch?v=0xC6tmtyDr8)

Source quotes

[0:04] Dimitri Bianco: 拿着方榫死命往圆孔里锤 [0:04] Dimitri Bianco: 瞄准靶心，而非好高骛远（Shoot for the Target, Not the Stars） [0:04] Dimitri Bianco: 即便把东西硬做出来了，这也是极其糟糕的 ROI [2:17] Dimitri Bianco: Dimitri，快来给我们做 AI [6:03] Dimitri Bianco: 悄无声息地死掉（Projects die quietly） [7:42] 外部供应商: 我们做 AI（We do AI） [14:14] Dimitri Bianco: 如果你没有合适的数据，就绝不可能构建出合适的模型 [16:09] Dimitri Bianco: 大家不看这些书，是因为我们根本没有时间！ [16:09] Dimitri Bianco: 试想一下，你招募了一个 2010 年毕业的员工，指望他在 2025 年掌握大语言模型（LLM）和生成式 AI（GenAI）的所有前沿技术，却完全不给他任何学习时间，这根本不切实际。 [17:46] Dimitri Bianco: 先学会爬，再学会走，接着慢跑，最后才能全速冲刺（Learn to crawl, walk, jog, and then run）

Source quotes

[0:04] Dimitri Bianco: Shoot for the target, not for the stars. [1:55] Dimitri Bianco: build me AI [4:12] Dimitri Bianco: improve customer experience [6:03] Dimitri Bianco: die quietly [7:42] Vendor Representative: We do AI. [13:36] Dimitri Bianco: I hate agile. I'm sorry if you're in tech. I can't do it. [14:14] Corporate Leadership: Why isn't this an AI solution? Why isn't this an XGBoost model? [14:14] Dimitri Bianco: You can't expect a home builder to build a house if they don't have the materials.

Original transcript

[0:04] Uh just quick introduction. I'm Demetri [0:05] Biano. Um I'm going to use the word [0:07] quant here quite a bit in the [0:09] presentation. I apologize. I'm in [0:11] quantitative finance. So you think back [0:13] to like you know I don't know like the [0:16] movies and stuff of all the traders on [0:17] Wall Street and they bring in their [0:19] quant and stuff. We're the people that [0:20] build the math, the stats, the models. [0:23] In finance typically we just aggregate [0:25] all this. So if you're doing math, [0:26] stats, research, ML, think in your head, [0:29] ML engineer, AI engineer, that's really [0:31] what I'm talking about throughout this, [0:35] but I'd like you guys to take a step [0:37] back because we keep talking about LLMs [0:39] the entire conference it feels like, but [0:41] AI is such a much bigger space and [0:43] there's a lot of other tools I feel like [0:45] we are missing as a whole. And hence the [0:47] title, shoot for the target, not for the [0:49] stars. Uh it does feel like as I go to [0:52] presentation after presentation we have [0:54] a round hole and we keep hammering the [0:56] square peg into it wanting it to fit. Um [0:59] and of course there are great [1:00] applications for LLMs but it's not going [1:02] to solve everything you have. Um [1:04] standard tech project 25 to 50% of these [1:06] fail right so I've worked at companies [1:09] have been the head of quantitative risk [1:10] and research I've worked at big global [1:12] banks national banks u working on [1:14] quantitative finance and risk management [1:16] and I can vouch that we see lots of tech [1:18] projects fail. Um there's really not a [1:20] lot of differences between why they fail [1:22] and why AI projects fail, but 70 to 85% [1:28] um of AI projects actually do fail. And [1:30] I want you to think about this not [1:32] necessarily of like, oh, we just [1:33] couldn't get it across the finish line. [1:34] This is return on investment. So if I've [1:37] got a project, I've got a deadline, I've [1:39] got, I don't know, half million dollars [1:40] to complete a project and we complete it [1:42] with $2 million. [1:44] I mean, you you made the project [1:46] complete, but it's a really bad ROI [1:48] here. So, a lot of these projects fail [1:50] and we're going to talk about why are [1:51] they failing? Like what are the key [1:53] issues here? Um, it's really going to be [1:55] planning, communication, data, and [1:59] skills. I know this seems simple, but as [2:01] we go through these, I'll give you guys [2:02] some anecdotal stories on why these are [2:04] not simple. I feel like many of the [2:06] presentations, right, it says like we [2:08] make this beautiful plan and it works [2:10] really well and it builds out this [2:12] solution and we just implement it. Uh, [2:13] unfortunately, that's not really what [2:15] happens. Um, so let's dive in here to [2:17] planning. Um, proper planning will give [2:20] you success and I have seen this at firm [2:22] after firm after firm. Um, I've been [2:25] around on the rise of ML in the banking [2:27] community. I've been on the rise with [2:29] AI. Um, we continually have executives, [2:32] teams, leads. Somebody gets really [2:33] jazzed up and excited. They come to a [2:35] conference and they say, "Demetri, come [2:37] here and build me AI." And I'm like, [2:40] "Great. So, like what are we doing with [2:42] this? Like what's the project? Like tell [2:44] me the use case with this." Right? [2:45] there's no planning involved. Um or more [2:47] specifically, they don't have a good [2:49] goal. So, you know, what's the problem [2:51] we're solving? How big is that problem? [2:54] And I'll give an example on the next [2:56] slide. Uh and what defines success? So, [2:59] I've seen many many models get put into [3:00] production. My team's really excited. [3:03] They work well. They hit everything we [3:04] thought we were supposed to hit. And [3:06] then the business gripes and complains [3:07] that it doesn't work as they wanted it. [3:09] It wasn't a great, you know, project. [3:11] But again, the success metrics weren't [3:13] set at the beginning and so the [3:15] expectation kind of failed as we go [3:16] through this. Um, ownership is one of [3:19] those things that I see a lot of issues [3:21] with. Um, who owns the project? Who owns [3:23] the success metrics? Who owns, you know, [3:25] the implementation of this? There's a [3:27] lot of people that are involved in these [3:28] projects. It's not going to just be a [3:30] technical team or a tech department. And [3:32] finally, resources, right? What do you [3:35] need for time, people, and technology? [3:38] And I'll tell you this, if you're [3:39] planning any AI project, I will almost [3:41] guarantee you're going to miss the time [3:43] component. There's always a miss there. [3:45] There's always a hurdle. There's always [3:46] something unknown that we end up running [3:48] into. Um, so time, people, and [3:50] technology. Having the right people to [3:51] implement these as well is critical, [3:53] right? There's all these vendor products [3:55] we're talking about. Give me your data, [3:57] guys. I'll just plug it in here [3:58] magically into this platform. It will [4:00] solve all of your problems. And then as [4:02] you get halfway through this project, [4:03] you've already invested a bunch of [4:04] money. um you realize you don't have the [4:06] right technology or the right people and [4:09] you start to run into some issues with [4:10] this. So planning failure again going [4:13] back to that you know is the gold too [4:15] big. Um I sat at a huge firm we spent a [4:19] fortune of money before you know all [4:21] this broke out with LLMs. We were going [4:24] to automate model development which is [4:26] mainly like what my team does. We were [4:27] going to automate this. We're going to [4:29] get rid of it. It was just going to be [4:31] this super easy process. And they spent, [4:33] I think, five to seven years doing this, [4:35] fortunes of money. And at the very end [4:37] of it, the result was a CEO came in and [4:39] sat down and was pissed off at how much [4:41] money they had wasted on this team [4:43] because they were going to automate this [4:44] entire thing. And the issue with this is [4:47] the person that was automating this [4:48] process thought model development was [4:50] like a one simple step. They didn't [4:52] realize this was a full entire process [4:54] of this. And so I have friends at some [4:56] of these large banks now and they're [4:57] looking at bite-sized pieces, right? So, [5:00] I'm sitting down. I'm coding out a [5:01] thousand 2,000 lines of code, writing a [5:03] lot of statistical tests and analysis, [5:05] looking at the dimensionality we have, [5:07] looking at the number space, trying to [5:08] figure out how we're going to solve this [5:10] problem. And they came up with and said, [5:12] you know what it is? We're going to work [5:13] on the documentation because even after [5:16] I build this entire model, I go through [5:17] this entire process, I still have to [5:19] write out documentation. And banks are [5:21] very, very highly regulated. Uh, model [5:23] documentation can be 70 80 pages long. [5:26] So after your quants, your ML engineers [5:28] build out all these models, build these [5:30] solutions, they have to sit down and [5:31] document all this. And even if you do [5:33] it, I encourage my employees to go step [5:34] by step throughout the process. You [5:36] know, make sure you document as you go. [5:38] They're using LLM now in this case to [5:40] simplify that because you make a few [5:41] notes, you put it in, and at the very [5:42] end, you just feed all this in, puts in [5:44] the correct format, the correct [5:45] template. Somebody goes through at the [5:47] very end, they read through the process, [5:48] and it simplifies it. again setting a [5:50] realistic goal instead of trying to say [5:52] like we're going to you know we're going [5:54] to improve customer experience like [5:56] that's not a goal that that's like a pie [5:58] in the sky dream but you don't have [6:00] understanding of the actual problem [6:01] itself [6:03] ownership is another big one I know this [6:05] shocks people but a lot of times I've [6:07] been at firms where there's actually no [6:09] ownership and every time there's a bump [6:12] or an issue or a hurdle it starts to [6:14] generate feuding and fights right how [6:17] come you didn't give me good data how [6:19] come the IT infrastru structure is not [6:20] stellar. How come, you know, the results [6:22] don't work? How come the model [6:23] development team's so far behind? And [6:25] it's not because you have an actual [6:27] hurdle or an actual issue. It's just the [6:29] responsibilities get dropped. The [6:31] ownership part of the planning doesn't [6:33] actually happen. And then when you get [6:35] halfway down the road, you end up [6:36] wasting months on these projects and [6:38] having more meetings than you need to [6:39] have. And you have employee burnout and [6:41] you lose a lot of the investment that [6:42] you put in. The other one which I was [6:45] shocked to see but as I've talked to [6:47] people around the industry projects die [6:49] quietly sometimes right so you got the [6:51] guy in the front he's really excited [6:53] he's going to build this AI project out [6:55] and he gets all excited his team gets [6:57] all together and then they reach out to [6:59] a data department they say hey we need [7:01] this data the data department goes it's [7:03] going to be an extra three four months [7:05] so data starts working on it priorities [7:07] in the company happen the project gets [7:09] shuffled off the data team drops the [7:12] project your quant team or your ML [7:14] engineering team all of a sudden dep [7:17] prioritizes it because it's been so long [7:18] they don't know if it's coming down the [7:19] pipeline or not and then all of a sudden [7:21] you're sitting in a board meeting and [7:22] you're asked about this project that you [7:24] were supposed to deliver and they're [7:25] like what's the progress on this and [7:27] you're like well the project actually [7:29] died right but you can't say that you [7:31] have to come up with an excuse and a [7:32] reason and try to revamp it um but again [7:34] having ownership on who owns what part [7:36] and keeping on top of that even from a [7:38] project management perspective uh will [7:39] help prevent some of these failures um [7:42] and then finally resour ources. So, I'm [7:44] going to say don't overvalue your vendor [7:46] tools. So, as I'm walking around the [7:48] conference here, I'm hearing every [7:49] vendor tool is going to solve every [7:50] problem I have. I even went to a firm [7:52] and I they said, you know, we could help [7:54] you solve your problems. I'm like, [7:56] great. I'm an independent. I don't even [7:57] run a firm anymore. Um they're like, [7:59] "No, no, no. We can solve your problem." [8:00] I said, "Well, great. Tell me what you [8:02] do." And the response was, "We do AI." [8:06] I'm like, "So, what does that mean?" And [8:08] they go, "Well, we can sit down with [8:09] you. We can look at your data. We can [8:10] look at your business. We can tell you [8:11] which problems you have." and then we [8:13] can build a solution. This sounds like a [8:15] really, really expensive consulting gig [8:17] is what this sounds like to me, right? I [8:18] mean, you're spending how many hours for [8:20] them? Tell me what's wrong with me. [8:21] That's not how this works. Um, again, [8:24] vendor tools are a nightmare as well [8:26] often because you're trying to patch and [8:27] put things together, right? So, I have [8:29] the world's best data management system, [8:31] the best infrastructure, and the best [8:33] project tool, and I'm trying to merge [8:34] these things together. Often, they don't [8:36] work quite well. So, be careful when you [8:38] go down the vendor route. Obviously, [8:39] you're going to need vendors, especially [8:40] if you're just getting going. you need [8:42] to get your feet, you know, your legs [8:43] underneath you to get going here. Um, [8:46] but be careful on that process and be [8:48] careful on the resources. Keep in mind [8:50] also self-learning takes a lot longer [8:53] than you expect, right? So, you're [8:55] really excited. We're going to do AI and [8:56] ML and you've got this team and all of a [8:59] sudden, you know, version five of a [9:01] model comes out like ChatGpt or [9:03] something and now we're like, wait, the [9:05] paradigm just shifted. You can't expect [9:07] the team just to like magically know [9:09] everything overnight and make that [9:10] update. So realize there's going to be [9:12] some self-learning. No matter where you [9:13] start in that process, no matter who you [9:15] hire, it's going to take time. And then [9:17] again, talk to all the departments. So [9:19] I've been on lots of projects and they [9:21] get like the model development teams, [9:22] they get the tech team in a room and we [9:24] forget that we need, you know, like a [9:26] the operations team to be there and now [9:29] we're building out this really cool [9:30] solution, but then when we go to pitch [9:31] it to operations, they tell us 10 [9:33] reasons why it doesn't work. And so we [9:34] just wasted a bunch of time on this [9:35] project as well. Um, so again, make sure [9:37] everyone is in the room that needs to be [9:39] there. Don't overkill it with 50 people, [9:41] but make sure you have the key [9:42] stakeholders there when you're putting [9:43] the resources together. And then [9:45] communication. So, there's a two-way [9:48] street here. I see a lot of people [9:50] complaining and saying the tech [9:51] departments, the techie people, they're [9:53] not telling me as the, you know, chief [9:56] executive or the COO, they're not [9:57] telling me the information I need. [9:59] They're not communicating very well. And [10:01] I'm like, well, great. So, are you [10:03] communicating to them very well? Right? [10:05] You have two parts here. you have very [10:07] technical people trying to explain to [10:09] very non-technical people an issue and [10:11] I've been there and I get going on this [10:12] 4-hour presentation and everyone's [10:14] looking at me like what the hell are you [10:15] talking about? Um, and then on the other [10:17] side of that, right, you have the [10:18] business who's explaining things crystal [10:20] clear to me and I'm sitting here like, I [10:23] don't quite understand why this is a [10:24] needed feature or why this is going to [10:26] help a customer. Um, but doing that [10:28] communication and sitting down and [10:30] making sure everybody really understands [10:31] the project before we get going. I know [10:34] I can kind of smile and nod sometimes [10:36] and like let's get this thing going. [10:37] Just give me the requirements. Um but [10:39] asking questions and improving your [10:40] communication is critical. Common [10:42] cultures or hurdles you're going to run [10:44] into with this is going to be data [10:45] infrastructure models, assumptions, and [10:47] skills. Uh the presentation previous to [10:49] this like Nvidia talking about this, [10:51] right? Do you have the right [10:52] infrastructure? I see a lot of firms [10:54] that are like, "Yeah, yeah, we're using [10:55] AWS or we're using a cloud or we have [10:57] everything. We know we've got it." But [10:59] you might not have the infrastructure [11:00] you need to actually implement the [11:01] solutions you want. Uh and again when [11:04] you're halfway through a project or [11:05] you're at the beginning and all of a [11:06] sudden now I don't have everything I [11:08] need. Now we're delaying weeks and [11:09] months because we all know the meetings [11:11] are going to get scheduled. Then you got [11:12] to talk to the vendor and there's a [11:14] bunch of other issues going on. Um so [11:16] again data infrastructure models and [11:18] assumptions are critical as well right I [11:20] built you a model. We saw this in [11:21] 2007208 when the financial crisis hit. [11:24] Um it crashed and it blew up and we all [11:26] blamed everyone pointed fingers on who [11:28] was at fault. But there were a lot of [11:30] assumptions in the models that failed [11:32] that a lot of people knew but we didn't [11:34] care if we were chasing dollars and [11:35] again the communication piece between [11:37] all the different departments and people [11:38] wasn't happening. So that was a key out [11:41] output I guess of the 2007208 [11:43] financial crisis there. Um and then [11:46] again skills like I mentioned you might [11:48] have a bunch of employees and they're [11:49] jazzed and excited and they get going on [11:51] this project and then you realize [11:53] partway through they just don't have the [11:54] skills to complete it. So now if I have [11:57] to bring in a vendor, a consultant, have [11:58] to hire someone, this is going to take [12:00] more time and more money. And I've seen [12:02] this drag projects out um to the point [12:04] that somebody overpromised a skill set [12:06] and now the company's kind of sitting in [12:08] a poor position with a large investment. [12:11] And then culture, culture is absolutely [12:13] critical here. Um I had a company I was [12:16] at where we had deadlines and we had [12:18] actually three definitions of deadlines [12:20] and it made me very upset because nobody [12:23] was talking about the exact same thing. [12:24] We had one group that was saying a [12:26] deadline is like a nice to have date, [12:28] right? January 5th, it's done by then, [12:30] great. If not, don't worry about it. [12:32] It's going to be another date. And then [12:33] we had another team that was my camp of [12:36] do or die. We will hit January 5th or [12:38] whatever the date is. It has to be done. [12:39] That is the deadline. Like it's going to [12:42] be like that. That's what it means by [12:43] definition. And then we had another [12:44] group of people that was saying a [12:46] deadline is like a date that it needs to [12:48] be done by, but we can always just sit [12:50] and negotiate a new deadline as we get [12:52] closer. So again, you can imagine when [12:55] you have different departments and [12:56] different people and even different [12:57] executives on this project and now we [13:00] all have different definitions of [13:01] deadlines, you know why the project [13:02] failed because nobody knew what the [13:04] actual deadline was. We were just [13:06] putting a nice pretty number out there [13:07] and hoping people are going to hit it. [13:09] And then again, how failures are [13:11] treated. So this I hope you take away [13:12] from this presentation. [13:14] Imagine you hire a bunch of staff. You [13:16] bring in a bunch of people if you're new [13:18] especially to the space with AI and ML [13:20] and you set them all up and you're going [13:22] to build this project and everyone's [13:23] really excited and they're all behind [13:25] you and then all of a sudden the project [13:27] fails for some reason. You don't have [13:28] the money, you don't have the data, you [13:30] don't have the people, you don't have [13:31] you're missing a bunch of stuff. Now [13:33] when you say this project sucks and they [13:35] failed and your model development team [13:36] is awful, are those employees going to [13:38] stick around? [13:40] I can tell you they're not. And this is [13:42] probably going to be something I'll talk [13:43] about in a future slides here kind of at [13:44] the Facebook meta issue right now. [13:46] Right? You can bring a bunch of all [13:47] stars into a team, but you still have to [13:50] be able to get, you know, everyone on [13:51] board on what is the expectation, what [13:53] is the culture. Being at a bunch of [13:54] different firms, it's going to be a [13:56] challenge. And then again, meetings in [13:58] progress. I hate agile. I'm sorry if [14:00] you're in tech. Um, I can't do it. I [14:02] have seen great tech teams do agile and [14:04] they love it and it works and it's [14:06] stellar. Um, but for me, it just doesn't [14:08] work out. And then again, [14:09] acknowledgement, celebrations. How do [14:10] you treat people? How do you tell them [14:12] they've done a great job? [14:14] Um, data. So, data, data, data. We've [14:17] heard this a lot at this conference. If [14:18] you don't have the right data, you're [14:20] not going to have the right models. Um, [14:22] I can build you guys super super simple [14:25] models. I've done this at a firm. And I [14:27] came in and built them a logistic [14:28] regression to solve this magical problem [14:29] they had. And they were like, why isn't [14:32] this an AI solution? Why isn't this a xg [14:35] boost model? And again, it was they [14:37] didn't have the right data. We didn't [14:38] have enough data. We're going to have [14:39] quality data. So data is something [14:41] that's critical. Again, these are the [14:43] six sigma definitions of quality on here [14:44] with the accuracy, completeness, [14:46] consistency, timely, validity, [14:47] uniqueness, integrity, and reliability. [14:50] And I've luckily been able to sit on [14:51] data teams and hear the complaints and [14:53] the issues and the challenges with this. [14:55] But data is going to define what you can [14:57] and can't do. I can promise you there's [14:59] no AI machine out there. There's no [15:02] kernel density estimation procedure that [15:04] can generate you some sort of synthetic [15:06] data that can create the world's best [15:09] model, right? If you don't have the [15:10] data, you can't build. You can't expect [15:12] a home builder to build a house that [15:14] don't have the materials. Um, again, [15:16] sizes and everything. I hear this a lot [15:18] with the data part of this where we say, [15:19] "Oh, we've got 10 million records. We [15:21] have 100 million records. We got four [15:22] terabytes of data." Um, and at the end [15:25] of the day, it doesn't matter. It's all [15:26] the same. There's no unique information. [15:28] There's not quality within there. [15:30] And then finally, skills. So I view [15:32] everything as a pipeline. I realize this [15:35] is not a common perspective, but you [15:37] have data. Data goes in. You have data [15:39] engineering. You have your business that [15:41] defines the problem up front. They pass [15:43] it to data. You make sure you have your [15:44] data. You have your model development [15:45] teams that develop the models. You have [15:47] your model validation teams which [15:48] validate and ensure the models are [15:49] accurate and use your risk management [15:51] practices. And then you find your tech [15:53] and your implementation. And then you [15:54] have the business at the end that's [15:55] giving the feedback. It's a pipeline. [15:57] There's always going to be an iterative [15:59] process with this. Um, but again, the [16:01] pipeline should be there. You should [16:02] have ownership and who owns which part [16:04] of the pipeline and you can figure out [16:05] where your bottlenecks are and you can [16:07] fix these issues quite quickly. [16:09] Management as well, talent acquisition [16:11] and retention is huge. Um, I can't say [16:14] that enough. It is very, very hard to [16:16] keep top talent. And so, going to that [16:18] meta example as well, right? They're [16:20] going to bring in a bunch of people. [16:21] They're going to throw a bunch of money [16:22] at them. Are they going to stick? I [16:25] don't know. Right? they need a retention [16:27] plan on how you bring them in. Um, oddly [16:29] enough, for a lot of us that work on the [16:31] research side and the development side, [16:32] model development. Um, I like to do like [16:35] educational ventures and learning and [16:37] reading and new skill sets. It's hard [16:39] for a lot of times the business to [16:40] realize I my employees would much rather [16:43] have a fun exciting research project to [16:45] work on than an extra 5 $10,000 in comp. [16:48] So, I have employees that have stuck [16:49] with me for years and years at different [16:51] firms and I've gone because of this. We [16:53] have a fun, exciting culture and an [16:55] environment where we embrace learning. [16:57] And I can tell you it's not enough just [16:58] to buy books. So I had a company that [17:00] bought a bunch of books. They put them [17:01] all on a bookshelf and they said, "Good [17:03] luck." And then nobody read the books. [17:06] And they said, "Reach, you're not even [17:07] reading the books." And of course, I [17:08] open my backpack and I've got like six [17:09] textbooks I'm carrying around with me. [17:11] But I said, "No one reads these books [17:13] because we don't need time, right? We [17:14] are so excited to hit this deadline to [17:16] maximize ROA. There's no learning going [17:18] on. And now you're upset. you're yelling [17:20] and screaming at me that my employees [17:22] are unhappy and that we're not hitting [17:24] deadlines, but at the end of the day, [17:25] you're not giving them the resources [17:26] they need, right? I mean, as we're [17:29] talking about, you know, LLMs and Genai [17:31] and everything's progressing so fast. [17:33] Imagine you said, "Okay, we're going to [17:34] take someone out of school that [17:35] graduated in 2010 and we're going to [17:37] expect them to know everything now in [17:39] 2025, but they have zero time to learn, [17:41] right? You can't really expect too much [17:43] from them in that capacity." [17:46] And then finally putting all this [17:47] together to get a positive ROI here. You [17:51] need to learn to crawl. Then you need to [17:53] learn to walk. Then you need to learn to [17:55] jog. And then you need to learn to run. [17:58] The whole premise of this presentation [18:00] talk is I see too many people saying we [18:02] don't even know how to crawl, but I'm [18:04] going to hit the gas and we're going to [18:05] sprint and we're going to run a [18:06] marathon. We have absolutely no [18:07] training. We have no skills. We have [18:09] minimal talent within our pool here, but [18:12] we're going to hire a bunch of other [18:13] tools and we're going to beat the [18:14] competition. I can promise you you will [18:16] not. You need to make modeling as an [18:19] entire process within your business as a [18:22] core competency and that includes [18:23] governance that includes risk management [18:25] and that includes the technical side of [18:27] the modeling and the business to define [18:29] those problems here. So if you can get [18:30] all these pieces put together you will [18:32] hit a win. Um if not you probably will [18:34] have bad ROI on projects and I you'll [18:36] hit that stat of 70 to 85% of projects [18:38] that just never complete because you [18:40] can't get down these kind of fundamental [18:42] basics here. Anyways, [18:45] thank you guys for having me. That was [18:46] my presentation. [applause]