Georgian's AI Lab Cuts Agentic Workflow Hype For Classical ML

Georgian AI 实验室破除智能体工作流炒作，回归经典机器学习 构建生产级生成式 AI 需要扎实的支撑脚手架、稳健的评估框架以及经典机器学习模型，而非脆弱的智能体工作流。 David Tingle 分享了为 Georgian 投资组合中的 B2B SaaS 初创企业构建人工智能解决方案所获得的十条宝贵洞见。概览指出，尽管 AI 的概念验证构建能够迅速带来惊艳体验，但投入生产环境却面临切实的软件工程挑战，需要稳健的评估体系、经典机器学习技术和可重复的架构。此外，演讲强调应对脆弱的智能体工作流保持审慎，指出了将领域专家引入跨职能小组的必要性，并建议各团队看淡 AI 炒作，专注于交付切实的商业价值。 Georgian 是一家总部位于多伦多的成长型投资机构，管理着超过 50 亿美元资产并投资了约 45 家 B2B SaaS 企业，其内部技术团队 AI Lab 每季度免费与被投公司并肩交付约 6 个 AI 软件项目。 团队警示切勿被极短的“令人惊叹时间”（time to wow）迷惑：虽然用 ChatGPT 或低代码工具快速构建概念验证很轻松，但将真正的 MVP 推向生产环境仍会经历艰难的“绝望之谷”。 在架构与模型选择上，必须提前建立完善的评估与可观测体系，且 XGBoost、逻辑回归等传统机器学习方法在基础设施、可解释性及评估大模型表现方面依然不可或缺。 由于复杂性与随机性过高，当前的 AI Agent 本质上依然十分脆弱，往往结构良好的多步骤非 Agent 工作流就足以实现同样的效果。 行业已逐渐走出炒作周期，团队应首先质疑是否真的需要 AI，并通过让领域专家深入跨职能 Pod 紧密协作，将重心彻底回归到解决实际业务痛点与客户价值上。

Drawing on hands-on experience from Georgian's in-house AI Lab, Head of Platform Operations David Tingle outlines how moving generative AI from rapid prototypes to reliable production requires strict evaluation, classical machine learning, and a return to problem-first engineering. Bridging the Gap from Prototype to Production Georgian, a Toronto-based growth-stage investor founded in 2008, manages just over $5 billion across seven active funds and roughly 45 Series B and Series C B2B SaaS companies across the United States, Canada, Israel, the UK, and other regions. Without charging fees, its in-house AI Lab builds trust with portfolio CTOs to work directly inside their codebases, completing about six engagements per quarter—ranging from four-to-five-day hackathons and one-to-two-sprint projects to three-to-six-month collaborations. Having built machine learning systems alongside startups since the pre-LLM era of BERT models, Tingle warns teams to beware the "time to wow." While ChatGPT and low-code tools make proofs of concept deceptively fast, pushing a minimum viable product into production still triggers a difficult "trough of despair" that modern AI coding assistants have not eliminated. To survive the transition to production, Tingle emphasizes that engineering teams must prioritize upfront evaluation and systemic repeatability. Without establishing assessment harnesses, logging, observability, and testing structures from the outset, teams frequently run in circles. Furthermore, the strongest teams avoid isolated, one-off builds; instead, they design repeatable systems capable of launching multiple reliable models, comparing their performance, and scaling successful patterns to new use cases. Balancing LLMs, Classical ML, and Agentic Workflows Although large language models currently anchor 80 to 90 percent of the AI Lab's builds, Tingle argues that classical machine learning remains vital to the surrounding infrastructure. Traditional techniques such as XGBoost, logistic regression, and clustering are well-understood, highly interpretable, and relatively easy to deploy. Crucially, they provide the quantitative evaluation required to measure LLM performance beyond mere qualitative impressions. During prompt design, Tingle also advocates for forcing a structured chain of thought. Even when a model does not strictly require step-by-step guidance to function, human-machine co-design using chain of thought clarifies assumptions, exposes edge cases, and maps out system logic, paying dividends despite producing more verbose prompts. Tingle offers a similarly pragmatic assessment of AI agents, which Georgian began deploying into production workflows around August of the previous year. He observes that agents remain fundamentally fragile because their stochasticity and complexity are difficult to manage. Making them reliable requires heavy engineering scaffolding—including explicit goal setting, interruption handling, and memory design—and Tingle notes that a well-designed, multi-step non-agentic workflow often performs just as well without relying on industry buzzwords. Problem-First Strategy and Cross-Functional Pods Before writing any machine learning code, the AI Lab's starting point is questioning whether AI is necessary at all. Tingle urges organizations to rigorously evaluate whether simpler, non-AI solutions can resolve the core business pain point more efficiently. When an AI build is warranted, execution depends heavily on organizational structure. Rather than dividing work across isolated technical layers, Tingle recommends forming structured, cross-functional pods where product design, infrastructure, machine learning engineers, and domain experts co-own the user problem. Embedding subject matter experts—specifically the actual daily users of the product—directly into the development process is invaluable for rapidly building the domain knowledge needed to make the system effective. Tingle concludes that the software industry has moved past the initial generative AI hype wave. Executive leadership and customers no longer care about AI as a gimmick or buzzword; instead, internal teams and external clients demand reliable solutions that improve workflows and deliver faster insights. Shedding the distraction of the hype cycle has allowed Georgian's portfolio companies to return to first principles and focus entirely on genuine customer value.

投资机构 Georgian 的平台运营负责人 David Tingle 总结了其技术实验室在被投企业中部署生成式 AI 的实践经验，指出企业必须摆脱对原型展示速度的盲目乐观，通过严格的系统评估、传统机器学习与大模型的融合，以及回归业务本质来克服实际投产中的工程挑战。 跨越原型幻象：生产落地困境与评估体系 David Tingle 指出，开发团队常落入“过早惊艳”（time to wow）的误区。利用 ChatGPT 或低代码工具构建概念验证（POC）虽然极为迅速且容易引发团队兴奋，但将其转化为真正的最小可行产品（MVP）并推向生产环境时，团队往往会陷入进展停滞的“绝望低谷”。在这一阶段，即便有现代 AI 编程助手支持，交付可靠软件的核心工程难度依然未变。 另一个关键痛点在于评估机制的缺失。许多团队在没有搭建恰当评估测试基准的情况下匆忙构建方案，导致由于缺乏前期的日志记录、可观测性和系统性测试结构而在排错与迭代中原地打转。Tingle 强调，成熟团队应当致力于建立具备可重复性的系统工程，使其不仅能稳定上线多个模型，还能定量对比模型表现，并将成熟模式顺畅扩展至新的业务场景。 技术栈实用主义：传统机器学习与大模型的协同 在模型技术选型上，Tingle 指出传统机器学习技术在现代 AI 架构中依然不可或缺。尽管当前 Georgian 实验室 80% 至 90% 的新构建项目以大语言模型为核心支柱，但 XGBoost、逻辑回归以及聚类等传统方法依旧是底层与外围基础设施的关键组成部分。 相比前沿模型，传统机器学习方法具备成熟、易于部署以及解释性强的显著优势。此外，这些方法为大模型本身的输出评估提供了客观的量化工具，帮助团队突破仅凭主观定性印象来判断大模型效果的局限，确保整体技术方案兼顾灵活性与确定性。 系统设计权衡：思维链价值与脆弱的自主智能体 在提示词工程方面，Tingle 认为推行“思维链”（Chain of Thought）大有裨益。即便特定模型在不给复杂指令的情况下也能运转，在人机协同设计阶段强制引入结构化思维链，有助于团队厘清潜在假设、暴露边缘异常场景并清晰梳理系统逻辑。虽然这会导致提示词更加冗长，但能为后续的系统维护带来显著收益。 对于自前一年 8 月起在生产环境中部署的“智能体”（Agents），Tingle 提出了审慎观察。他认为当前智能体本质上十分脆弱，系统复杂性与随机性极难管理，需要庞大的支撑架构（包括目标设定、中断处理机制和记忆设计）。在许多实际场景中，设计良好的多步骤非智能体工作流完全能达到同等效果，且避免了因追逐时髦概念而引入不必要的系统脆弱性。 组织与战略协同：审视 AI 必要性并深耕业务价值 在战略决策层面，Georgian 实验室的起点是始终主动质疑“是否有必要使用 AI”。Tingle 敦促团队在立项前严格考量，若非 AI 的简单方案能更高效地解决业务核心痛点，就应优先采用简易方案。当确有必要引入 AI 时，必须让产品的真实日常用户——即业务领域专家（SME）——直接嵌入开发流程，以快速补齐行业知识。 这种协同依赖于跨职能的“Pod”组织架构：产品设计、底层基础架构、机器学习工程师与领域专家共同对用户痛点负责，打破传统割裂的职能壁垒。Tingle 总结称，当前行业已走出初期的狂热炒作周期，管理层与客户不再为概念噱头买单，而是聚焦于能切实优化工作流、加速洞察产出并稳定运行的真实价值。 David Tingle 的分享揭示了生成式 AI 落地从“追逐技术概念”向“深耕工程现实”的转变：企业成功投产的关键不在于追求前沿技术的复杂度，而在于通过可观测的工程闭环、严谨的跨职能协作，以及对非 AI 替代方案的理性权衡来保障交付确定性。

## Background 这段视频由 Ai4 于 2025 年 8 月 27 日发布在 YouTube 平台上。视频记录了 Georgian 平台运营负责人 David Tingle 的演讲，内容主要涵盖 Georgian 的 AI Lab 在跨多个垂直领域和使用场景构建及部署生成式 AI 解决方案时总结出的核心经验教训。 ## TL;DR 演讲围绕 Georgian 在协助旗下投资组合公司落地 AI 过程中积累的 10 条宝贵经验展开。David 深入探讨了警惕 PoC 阶段的“惊艳期陷阱”、建立完善评估测试机制的重要性，以及传统机器学习在当下不可替代的价值。同时，他还分享了关于代理工作流（Agents）的局限性、思维链的应用、跨职能团队结构搭建，以及如何摒弃技术炒作、重新回归解决客户实际问题与创造业务价值的第一性原理。 ## Atmosphere & style 整场演讲氛围专业而务实，又不失幽默风趣。演讲者以坦诚接地气的风格，将原本可能枯燥的技术与投资经验分享得深入浅出。既有对行业过度炒作的冷静反思，又有来自一线实操的深刻洞察，带给观众一种既清醒又极具启发性的聆听体验。 ## Key takeaways - Georgian 是一家总部位于加拿大多伦多的成长型投资机构，管理资产超 50 亿美元，主要投资 Series B 到 C 阶段的 B2B SaaS 企业。 - Georgian 内部拥有专属技术团队 AI Lab，每年以每季度约 6 个项目的节奏与投资组合公司合作，提供数周短平快项目、黑客马拉松及 3 到 6 个月的深度共建。 - 警惕“惊艳期”陷阱（Time to Wow）：PoC 和 Demo 阶段开发极快，但从 Demo 到真正的 MVP 和产品化落地会经历艰难的“绝望之谷”，需对落地难度保持理性认知。 - 评估机制（Evaluation）决定项目成败：必须尽早搭建围绕 AI/ML 的评估测试框架、日志记录与可观测性，避免盲目开发却无法评估成效。 - 经典机器学习（Classical ML）依然不可或缺：XGBoost、逻辑回归、聚类分析等传统技术更具可解释性且易于生产化，可为 LLM 架构提供坚实支撑。 - 追求可复用性（Repeatability）：优秀团队既注重在单一产品中构建支持多模型切换的稳定系统，也注重将经验高效扩展至不同业务用例。 - 思维链（Chain of Thought）在提示词设计中大有裨益：在人机共同设计中应用 CoT 有助于厘清假设、处理边缘情况并理顺逻辑，带来长期回报。 - 代理工作流（Agents）依然脆弱：Agent 系统因复杂性和随机性高而极度依赖外围脚手架；在不需要 Agent 噱头时，确定性的多步骤系统效果更好。 - 回归需求本质（Is it really necessary?）：解决问题前应先审视痛点与指标，优先考虑是否存在更简单、非 AI 的解决方案，避免为了用 AI 而用 AI。 - 深度融入领域专家（SME）：将懂业务且作为最终用户的领域专家直接嵌入开发流程，能显著提升代码质量并加速团队对业务的理解。 - 建立跨职能敏捷小组（Pods）：相比按技术层级划分的传统架构，凝聚产品设计、基础设施、机器学习与领域专家的战术小组更能产出高质量产品。 - 摒弃炒作，回归客户价值：客户不在乎底层技术细节，高管不再盲目追逐热词，专注于改善工作流、提升洞察准确度与效率才是核心。 ## Detailed walkthrough ### 简介与 Georgian 公司的投资背景 David Tingle 在演讲开场时首先进行了自我介绍，并幽默地提到自己的合规团队要求他发表免责声明：他所分享的内容并非投资建议或法律建议。接着，他介绍了自己所在的总部位于加拿大多伦多的成长型软件投资机构 **Georgian**。该公司成立于 2008 年，主要投资于 **Series B** 至 **Series C** 阶段的 B2B SaaS 企业。 * **资产规模与组合：** 管理资产规模刚刚超过 50 亿美元，横跨 7 支基金，拥有约 45 家活跃的投资组合公司。 * **投资地域分布：** 投资组合遍布美国、加拿大、以色列、英国及其他少数地区。 * **核心竞争力（AI Lab）：** 公司的“秘密武器”是其内部的技术团队——**AI Lab**。该团队的使命是与投资组合公司共同构建软件，核心垂直领域是 AI 和机器学习。在业界普遍开始热议 LLM（大语言模型）之前，他们就已经在实际应用 **BERT** 模型了。 AI Lab 的合作模式非常独特。尽管初创公司的 CTO 通常不会轻易让投资人接触核心代码库，而且这些公司并不需要向他们支付服务费，但 Georgian 依然成功建立了信任。他们每年以大约每季度 6 个项目的节奏与投资组合公司紧密合作。这些合作项目的周期长短不一： * 几周（一两个 sprint）的短平快项目。 * 持续 4 到 5 天的黑客马拉松（hackathons）。 * 标准的 3 到 6 个月深度合作项目，与组合公司的开发团队携手并肩。 --- ### 构建生成式 AI 的十条核心经验教训 David 指出，接下来的分享涵盖了他们在过去几年中构建 AI 时“来之不易的深刻见解”（hard-won insights）。虽然有些观点在团队内部准备幻灯片时曾引发争议甚至被认为颇具颠覆性，但他依然毫无保留地总结了 10 条宝贵的经验教训。 * **警惕惊艳期的陷阱（Beware the time to wow）：** 利用 ChatGPT 或开发 PoC（概念验证）时，速度极快且极其容易让人兴奋，仿佛产品即将腾飞。然而，真正的 MVP（最小可行性产品）和产品化落地却要经历艰难的“绝望之谷”（trough of despair）。虽然鼓励前期多尝试低代码或无 code 工具，但必须对落地的难度保持清醒和现实的认知。 * **评估机制至关重要（Evaluation really matters）：** 许多团队常犯的错误是“建完却无法评估成效”（chase your tail）。必须尽早搭建好围绕 AI 和 ML 的评估测试框架，涵盖日志记录（logging）和可观测性（observability）等工程最佳实践。虽然这部分不如写代码炫酷，但决定了项目的成败。 * **经典机器学习依然不可或缺（Classical ML still matters）：** 尽管生成式 AI 浪潮汹涌，且 LLM 占据了许多核心架构的 80% 到 90%，但传统的机器学习技术（如 **XGBoost**、逻辑回归、聚类分析）依然至关重要。它们不仅更容易投入生产且具可解释性，还能为 LLM 提供基础支撑，并帮助团队超越单纯的主观定性评估。 * **追求系统与用例的可复用性（Repeatability）：** 优秀的团队非常注重可复用性，具体体现在两个维度：首先是在单一产品或功能中，如何构建能够支持多个模型对比与切换的稳定系统；其次是如何将这种经验高效扩展到多个不同的业务用例中。正如加拿大经典喜剧《富家穷路》（Schitt's Creek）中的台词所表达的，优秀的机器学习团队必须具备“能够再次做到这一切”的坚定信念。 * **思维链（Chain of Thought）大有裨益：** 这里的思维链不仅指推理模型本身的工作方式，更强调在提示词设计（prompt design）中的应用。尽管模型本身可能并不严格需要它们，但在人机共同设计（human-machine co-design）的过程中，思维链有助于厘清假设、处理边缘情况并理顺逻辑。这往往会导致稍微有些冗长的提示词或系统，但在后续开发中会带来巨大的回报。 --- ### 从代理工作流、业务必要性到团队结构的进阶反思 * **代理工作流（Agents）依然脆弱，需谨慎权衡：** Georgian 从去年 8 月起便开始涉足真正的代理式工作流。目前的观察是：代理系统依然非常脆弱，因为复杂性和随机性（stochasticity）极难控制。因此，它们极度依赖目标设定、中断处理和内存设计等外围脚手架（scaffolding）。如果无法带来质的飞跃，有时采用确定性的多步骤系统反而比单纯追求“Agent”噱头能产出更好的效果。 * **回归本质：AI 真的非用不可吗？（Is it really necessary?）：** AI 实验室的初始冲动往往是“我们能不能用 AI 解决它？”。但最理智的起点应当是审视痛点本身：是否真的需要 AI？是否存在更简单、更不依赖 AI 的解决方案？如果能把痛点、影响和衡量指标梳理清楚，往往会发现很多问题根本无需复杂的 AI 系统就能迎刃而解。 * **深度融入领域专家（Subject Matter Experts）：** 传统的做法是将数据科学团队与机构其他部门完全隔离。然而，他们合作过的最优秀的项目，都是将既懂业务又是产品直接用户的领域专家（SME）直接嵌入到开发流程中。这不仅极大提升了代码质量，也加速了开发人员对业务领域的理解。 * **跨职能敏捷小组（Cross-functional Pods）：** 团队结构会直接影响最终推向市场的模型质量与产品水平。相较于传统的按技术层级划分的架构，结构化的跨职能“战术小组”（pods）——将产品设计、基础设施、机器学习和领域专家凝聚在一起、共同对用户问题负责——能够发挥出更强大的威力。 * **超越喧嚣的炒作，回归客户价值（Beyond the Hype）：** 如今，行业已经走过了盲目追逐炒作的初期阶段。客户根本不在乎底层用的是不是大语言模型（LLM），企业高管也不再盲目为了“搞 AI 而搞 AI”。真正让客户满意的是工作流变得更顺畅、洞察更准确、效率更高。摒弃浮华的 AI 术语和炒作，回归以客户为中心的第一性原理，才是当前最健康、最高效的发展状态。 By the way, to unlock the full functionality of all Apps, enable [Gemini Apps Activity](https://myactivity.google.com/product/gemini). ## Notable quotes > 警惕惊艳期的陷阱（Beware the time to wow）：利用 ChatGPT 或开发 PoC（概念验证）时，速度极快且极其容易让人兴奋，仿佛产品即将腾飞。然而，真正的 MVP（最小可行性产品）和产品化落地却要经历艰难的“绝望之谷”（trough of despair）。 — David Tingle > 评估机制至关重要（Evaluation really matters）：许多团队常犯的错误是“建完却无法评估成效”（chase your tail）。必须尽早搭建好围绕 AI 和 ML 的评估测试框架，涵盖日志记录（logging）和可观测性（observability）等工程最佳实践。 — David Tingle > 经典机器学习依然不可或缺（Classical ML still matters）：尽管生成式 AI 浪潮汹涌，且 LLM 占据了许多核心架构的 80% 到 90%，但传统的机器学习技术（如 XGBoost、逻辑回归、聚类分析）依然至关重要。 — David Tingle > 追求系统与用例的可复用性（Repeatability）：优秀的团队非常注重可复用性，具体体现在两个维度：首先是在单一产品或功能中，如何构建能够支持多个模型对比与切换的稳定系统；其次是如何将这种经验高效扩展到多个不同的业务用例中。 — David Tingle > 思维链（Chain of Thought）大有裨益：这里的思维链不仅指推理模型本身的工作方式，更强调在提示词设计（prompt design）中的应用。 — David Tingle > 代理工作流（Agents）依然脆弱，需谨慎权衡：目前的观察是：代理系统依然非常脆弱，因为复杂性和随机性（stochasticity）极难控制。 — David Tingle > 回归本质：AI 真的非用不可吗？（Is it really necessary?）：最理智的起点应当是审视痛点本身：是否真的需要 AI？是否存在更简单、更不依赖 AI 的解决方案？ — David Tingle > 深度融入领域专家（Subject Matter Experts）：他们合作过的最优秀的项目，都是将既懂业务又是产品直接用户的领域专家（SME）直接嵌入到开发流程中。 — David Tingle > 超越喧嚣的炒作，回归客户价值（Beyond the Hype）：摒弃浮华的 AI 术语和炒作，回归以客户为中心的第一性原理，才是当前最健康、最高效的发展状态。 — David Tingle ## Mentioned David Tingle, Georgian, AI Lab, Ai4, BERT, ChatGPT, XGBoost, LLM, Schitt's Creek ## Video description David Tingle, Head of Platform Operations, Georgian Abstracts Georgian's AI Lab works with Georgian's portfolio of software scaleups to develop and deploy AI solutions across a variety of verticals and use cases. David's talk will cover some of the most important lessons learned across this work, covering trends we've seen, design & development patterns, how to drive business impact, and key factors that impact the overall success or failure of an AI initiative [Source](https://www.youtube.com/watch?v=giOa1dJ27KI)

David Tingle: 警惕惊艳期的陷阱（Beware the time to wow）：利用 ChatGPT 或开发 PoC（概念验证）时，速度极快且极其容易让人兴奋，仿佛产品即将腾飞。然而，真正的 MVP（最小可行性产品）和产品化落地却要经历艰难的“绝望之谷”（trough of despair）。 David Tingle: 评估机制至关重要（Evaluation really matters）：许多团队常犯的错误是“建完却无法评估成效”（chase your tail）。必须尽早搭建好围绕 AI 和 ML 的评估测试框架，涵盖日志记录（logging）和可观测性（observability）等工程最佳实践。 David Tingle: 经典机器学习依然不可或缺（Classical ML still matters）：尽管生成式 AI 浪潮汹涌，且 LLM 占据了许多核心架构的 80% 到 90%，但传统的机器学习技术（如 XGBoost、逻辑回归、聚类分析）依然至关重要。 David Tingle: 追求系统与用例的可复用性（Repeatability）：优秀的团队非常注重可复用性，具体体现在两个维度：首先是在单一产品或功能中，如何构建能够支持多个模型对比与切换的稳定系统；其次是如何将这种经验高效扩展到多个不同的业务用例中。 David Tingle: 思维链（Chain of Thought）大有裨益：这里的思维链不仅指推理模型本身的工作方式，更强调在提示词设计（prompt design）中的应用。 David Tingle: 代理工作流（Agents）依然脆弱，需谨慎权衡：目前的观察是：代理系统依然非常脆弱，因为复杂性和随机性（stochasticity）极难控制。 David Tingle: 回归本质：AI 真的非用不可吗？（Is it really necessary?）：最理智的起点应当是审视痛点本身：是否真的需要 AI？是否存在更简单、更不依赖 AI 的解决方案？ David Tingle: 深度融入领域专家（Subject Matter Experts）：他们合作过的最优秀的项目，都是将既懂业务又是产品直接用户的领域专家（SME）直接嵌入到开发流程中。 David Tingle: 超越喧嚣的炒作，回归客户价值（Beyond the Hype）：摒弃浮华的 AI 术语和炒作，回归以客户为中心的第一性原理，才是当前最健康、最高效的发展状态。

David Tingle: secret sauce David Tingle: beware the time to wow David Tingle: trough of despair David Tingle: classical machine learning still matters David Tingle: can do it again David Tingle: chain of thought is good for you

[0:04] My name is David as our MC introduced. [0:07] Uh I work at a company called Georgian. [0:08] We'll go through that in a minute. One [0:10] of the details about Georgian is that we [0:12] are primarily an investor. And so my [0:15] compliance folks make me not necessarily [0:18] read this out, but at least articulate [0:20] that anything I share is not investment [0:22] advice. It's not legal advice, etc., [0:24] etc. You get the picture. Okay. So now [0:26] we can move on. [0:28] So what is Georgian? [0:30] Georgian is a growth stage software [0:32] investor based in Toronto, Canada. Uh [0:35] we've been around since 2008 and we [0:38] invest in B2B SAS businesses uh usually [0:41] at the series B to series C stage. We [0:44] have an active portfolio of about 45 [0:46] companies and uh just under just over [0:51] five billion in assets under management [0:53] across seven funds. So fairly active. [0:55] The portfolio is spread across the US, [0:57] Canada, uh, Israel, the UK, and a couple [1:00] of other places. Our secret sauce as a [1:03] fund, because every fund needs a secret [1:05] sauce, right? Writing a check is easy. [1:07] Our secret sauce as a fund is that we [1:08] have an in-house technical team, the AI [1:11] lab. And our mandate, the team that I [1:13] work with, our mandate is primarily to [1:15] build software with our portfolio [1:17] companies. And our niche is AI and ML. [1:19] And we've been doing this for several [1:20] years. We were implementing BERT models [1:23] before LLMs were what people called [1:25] LLMs. Um, that takes a couple different [1:28] flavors. So, we'll do advisory work, but [1:31] what we really like doing is getting [1:32] hands-on in the development process and [1:35] identifying the things that our [1:36] portfolio companies want to build that [1:38] we can help them with. It's a whole [1:40] another conversation how we build the [1:42] trust with them to allow us to help them [1:43] build because they don't pay us. And if [1:46] I'm a CTO of a fast growing startups, [1:49] I'm not going to let my investor into my [1:50] codebase. But we do that. We we manage [1:53] to work with portfolio companies at a [1:54] rate of about six projects per quarter. [1:57] Um some of those are a few weeks, a [1:59] sprint or two. Some of those are [2:02] hackathons that we run over four or five [2:04] days. But the standard model is kind of [2:06] a three to six month engagement where [2:07] we're working handinhand with the [2:08] development team. And to enable that, we [2:11] have a team of machine learning [2:12] researchers and engineers um who are [2:15] some of the best people that I've ever [2:16] worked with. It's a small team, but we [2:18] managed to do a lot of good work. And [2:21] everything I'm sharing in this talk is [2:22] kind of hard one insights from the [2:24] process of building AI over the last few [2:26] years. So, let's get into it. As I said, [2:30] Georgian builds with the portfolio. So, [2:33] what have we learned in the process? And [2:34] hopefully some of those learnings are [2:36] useful to you. I will say right off the [2:39] bat that some of them are probably [2:41] pretty controversial too. I know there [2:42] was a debate as I was putting these [2:43] slides together. Some people were like, [2:44] "Ah, you shouldn't put this in there or [2:46] this doesn't make sense." So, take this [2:48] as kind of a perspective. It might not [2:50] be the right perspective all the time. [2:51] It might not be your perspective, but um [2:53] it does share some of the trends that [2:55] we've been seeing as we work across the [2:57] portfolio. [3:00] So, number one big lesson for us is this [3:02] idea of beware the time to wow. You [3:05] might have already experienced this if [3:06] you're developing PC's. It's very easy [3:08] to get to something that kind of works, [3:11] right? Even using chat GPT. Um, and [3:14] often that first PC will be super fast [3:16] and get everyone really excited and you [3:18] want to move very fast. A real MVP [3:21] getting stuff into production takes [3:23] time. This is very hard one lesson for [3:25] us. I got a simple graph here. [3:27] Everyone's really happy right at the [3:28] beginning of the the uh the time on the [3:30] left side of the X- axis. uh then we get [3:33] into this trough of despair where it's [3:35] like actually shipping stuff is hard. it [3:37] doesn't change in the IML era despite [3:39] all the advancements in AI coding and [3:41] then ideally you get to this rocket ship [3:43] stage where the product takes off but [3:45] it's never easy okay so be aware of this [3:47] be realistic you're doing that doesn't [3:49] mean you don't build PC's in fact we [3:52] encourage all of our portfolio companies [3:54] to try stuff out in chat GPT to use no [3:56] code or low code workflows before [3:57] starting to invest time in development [3:59] cycles but be aware it's harder than it [4:02] looks [4:03] second lesson is evaluation really [4:05] matters [4:06] And that that's not as exciting a title, [4:09] but it really matters for us. Um, one of [4:13] the patterns that we've fallen into [4:15] ourselves and we've seen portfolio [4:16] companies fall into when we work with [4:17] them is building and then not being able [4:21] to assess the impact of what we're [4:22] shipping. And so we try to advise [4:24] everyone not to chase your tail, right? [4:27] Figure out the evaluation harness that [4:28] you have around the AI and ML that you [4:30] want to build early on in the process, [4:32] what whatever that means in your [4:34] specific context. logging, [4:36] observability, all the tools that make [4:38] good engineering practices possible. Um, [4:40] think about that upfront and evaluation [4:42] is very very critical to get right. It's [4:44] not the coolest part of what you're [4:46] building, but it's really critical and [4:48] this is a hard one lesson for us. [4:51] Third one is that classical ML really [4:54] still matters. The backbone of our team [4:55] is kind of uh language um natural [4:58] language processing and NLG. Um and a [5:02] number of the team members would still [5:03] classify themselves as researchers [5:05] first. Um and their perspective is that [5:08] even in this generative AI wave when LM [5:10] are at the core of a lot of what we're [5:12] building probably 80 to 90% now [5:14] classical ML techniques like XG boost [5:16] logistic regression clustering those [5:18] things are still critical and they form [5:21] part of the infrastructure around the [5:23] LLM. Um these things are understood [5:27] they're easy to put into production [5:28] relatively. They're interpretable. You [5:31] can build systems around them and [5:33] they're very helpful when you want to [5:35] evaluate how your LMS are performing and [5:37] you want to go beyond a qualitative [5:38] assessment. So classical ML still [5:41] matters. This is somewhat related. Okay. [5:47] Third thing, you're kind of sensing a [5:49] key here. The best teams we've worked [5:51] with think about repeatability a lot. [5:54] And they think about repeatability in [5:55] two ways. one they're thinking about how [5:58] do we build so that we can launch [6:01] reliable models multiple models we can [6:03] compare and contrast the performance [6:04] across multiple models and we can build [6:07] a really good system so even in the [6:08] context of a single product or a single [6:10] feature how are we building [6:12] repeatability and reliability into the [6:14] system but then second how do we scale [6:17] that to multiple use cases efficiently [6:19] right if it's good for one product or [6:20] one feature how do we extend that and [6:22] build the infrastructure so that's [6:23] really easy to do across multiple use [6:25] cases [6:26] And this isn't, I would say, a solved [6:28] problem. We still struggle with this. [6:30] Um, but thinking with that principle in [6:34] mind of repeatability and scalability is [6:37] really valuable and the best teams we've [6:38] worked with think about this a lot. This [6:40] is Shitz Creek, which is a Canadian TV [6:42] show. I don't know if you have it down [6:43] here, but this person is very confident [6:46] that they can do it again. And that's [6:47] what all good ML teams do these days. [6:51] Next point. And I've lost I have 10. I [6:53] have 10. Okay. So, keep me on track. But [6:56] I don't know what number we're on now, [6:57] so we'll see. I think four. Chain of [6:59] thought is good for you. Uh, so chain of [7:02] thought is can mean a couple different [7:04] things. It definitely means kind of how [7:06] models how reasoning models work through [7:08] things, but it also means something in [7:10] terms of prompt design. And that's [7:11] primarily what I mean it in terms of [7:13] today. So models might not strictly [7:15] needed, right? You might get great [7:16] performance out of very simple [7:18] instructions from the model. But what we [7:20] found is that chain of thought is really [7:21] useful even in the process of building [7:24] systems. It might not be fundamentally [7:26] necessary for the AI to solve the [7:29] problem or get where it wants to go, but [7:32] it's very useful for understanding what [7:33] you want the model to do, clarifying [7:35] assumptions, edge cases, thinking [7:37] through the reasoning and the logic of [7:38] what you're trying to do. So we [7:40] generally think that from a human [7:43] machine codeesign perspective chain of [7:45] thought is good practice and it might [7:48] result in kind of maybe slightly over [7:50] overly verbose prompts or systems but [7:52] often that is pay that pays dividends [7:55] down the road. Mattie Mat another great [7:58] Canadian he says it's good for you so do [8:01] it. [8:03] Okay point here uh agents we do we do a [8:06] fair bit of agent work now. I think we [8:09] shipped our first kind of proper agentic [8:12] workflow with a portfolio company in [8:15] August of last year. Um, and it's [8:18] something that we're actively working on [8:19] in a number of projects. And the [8:21] observation currently, and this may [8:23] evolve, this will evolve, but the [8:25] observation currently is that agents are [8:26] still really fragile, right? Complexity [8:28] is hard. Um, stochasticity is difficult. [8:32] Um, so when you're building agentic [8:33] systems, our experience is they really [8:35] do benefit from things like goal [8:37] setting, interruption handling, memory [8:40] design, all the things that you might [8:42] not necessarily want to build, but we [8:44] find um makes a big impact. And a [8:48] broader learning here is that often the [8:51] alternative to an agentic workflow um [8:54] can be just as good, right? Um [8:57] multi-step systems where it's not truly [8:59] agentic, right? you're not getting that [9:00] buzzword uh um kick, that dopamine hit, [9:05] but they're still super super valuable [9:07] and often they outperform kind of a a [9:10] stateless agent in a lot of the use [9:12] cases that we're seeing. That'll change [9:13] over time, right, as this technology [9:15] updates, but today we're seeing this [9:18] even when you're building even when [9:19] we're building aic systems. The [9:22] observation from the team is that the um [9:25] the uh ar the um what's the language I'm [9:29] looking for? The architecture the [9:31] scaffolding around the agent is really [9:33] important and that can be deterministic [9:35] but if it's welldesigned it can really [9:37] benefit guiding the agent to the proper [9:39] task and a proper result. [9:42] This is a big one and maybe I'm getting [9:44] into bigger themes now. We're an AI lab. [9:47] We want to build AI solutions. We like [9:49] doing machine learning work. Ultimately, [9:51] we want to work with our companies, but [9:53] the starting point for us always is, is [9:55] it really necessary? Like, do we have to [9:57] do this? Are there better ways to do it [9:58] or simpler ways to do it that aren't AI? [10:00] And the starting point in the [10:02] conversation is that, you know, you [10:04] might have this problem that you've [10:05] identified before getting to AI is the [10:08] solution. Try to evaluate a broader set. [10:10] Really focus on what the pain point is [10:12] or what the opportunity that you see is. [10:14] If you can be really crisp on what [10:16] you're addressing, why it matters, and [10:18] how you would measure what the impact [10:21] is, then that allows you to assess, [10:23] okay, do I invest a lot of effort in an [10:24] AI solution, or is there something [10:26] simpler that could address these pain [10:27] points in a different way, a different [10:29] process, different technology? [10:31] Ultimately, we want to, you know, we [10:32] want to do good work and we're going to [10:34] there'll be plenty of opportunities for [10:36] us to collaborate on the AI stuff, but [10:38] we feel really good when we early on in [10:40] the process work with a company and say [10:42] actually this would be this is simpler [10:44] to solve, right? You don't need a [10:45] complex system. You can solve this with [10:48] uh a different solution. [10:51] Big learning here, subject matter [10:53] experts are the best, right? Um and I [10:56] think this is probably shifting but [10:58] there was a world where like data [10:59] science was totally carved off from the [11:01] rest of the institution and uh talked to [11:04] subject matter experts hopefully that's [11:06] changing for better. We've seen the best [11:08] projects we work on we have somebody [11:10] who's a subject matter expert and not a [11:12] data scientist or a machine learning [11:14] engineer or research scientist directly [11:15] embedded in that development process. Um [11:18] we always try to build and to involve [11:21] the folks who are the users of the [11:23] product. doesn't always work out that [11:25] way, but if we can get them involved in [11:27] the code-development process, that to us [11:29] is very very impactful on the quality of [11:32] the work and also the speed at which we [11:34] can ramp up the domain knowledge of the [11:36] actual developers. So there's a teamwork [11:39] gif here because you know teamwork makes [11:41] the dream work. [11:43] Another team one, the team structure in [11:46] terms of how you're developing um really [11:48] does impact the model quality of the [11:49] product that you're ending up uh putting [11:51] out into market. We found that the kind [11:54] of like structured cross functional pods [11:57] especially in AI use cases work really [11:59] well and they outperform kind of a more [12:03] traditional tech layering effect. Um and [12:08] we we want in general [12:11] all stakeholders all parts of the [12:12] development process to kind of co-own [12:14] the user problem. So the degree to which [12:16] you can bring different teams together, [12:18] product design, infrastructure, ML, [12:21] subject matter experts, I think it it [12:23] really does pay dividends. That's been [12:24] our experience in developing [12:27] another [12:28] I should have found a different gift. I [12:30] don't know. But teamwork does make the [12:31] dream work. [12:33] Big one here and I think this is [12:34] shifting our guidance. Maybe my guidance [12:38] to you or our perspective now is that [12:40] like we're we're a little bit past the [12:41] hype wave here. Um, in all the stuff [12:44] that we're working on today, [12:47] I think there's a recognition with the [12:49] teams that we're working on that like it [12:50] doesn't re like the customer doesn't [12:52] really care if it's AI and like the AI [12:54] selling point doesn't matter as much [12:57] anymore. We're past that. And also the [12:59] motivation from executive leadership [13:01] isn't we need to build AI for the sake [13:03] of building AI to be part of this. We're [13:05] in a new world, I think, or we're moving [13:07] into a new world where we've really [13:09] gotten back to kind of first principles [13:10] and customers are articulating that. [13:12] That's our experience. They want real [13:14] solutions. They don't want fluff. They [13:16] don't want AI jargon or hype. Um, [13:19] ultimately, no, I think nobody cares is [13:21] a strong statement. Nobody cares whether [13:22] it's an LLM or whatever it is under the [13:24] hood solving the problem. If their [13:26] workflow is better, if they get better [13:28] insights, if stuff gets done faster or [13:30] more in a more way, that's what make c [13:32] that's what makes customers happy. [13:34] Customers here, internal teams, external [13:37] customers, whatever you want to want to [13:38] do here. But the hype I think detracted [13:43] in the early stages of our work from the [13:45] ability to do do good kind of customer [13:47] focused development work. And we're past [13:49] that now. That's our experience. It's [13:51] been really really productive for the [13:53] team and for the groups that we work [13:54] with to kind of refocus on customer [13:57] problems and be a little bit um kind of [14:00] indifferent to catching a wave or [14:03] playing into the hype and trying to [14:04] articul be a part of that journey. [14:08] Those are the 10 lessons we've learned [14:11] shipping across the portfolio. Um, if [14:13] this is interesting, I'd be happy to [14:14] chat afterwards. Um, our model is a [14:17] little bit different for an investor. [14:18] Um, so we have a slightly different [14:20] perspective. It might differ from how [14:21] you approach things, but in general, [14:24] hopefully you found this useful. Um, [14:26] we'll have 10 new ones next time I come [14:28] back here. Thank you all very much for [14:30] your time.