Learning Fast from AI Missteps: What Actually Worked in the Real World With Cushman & Wakefield

从 AI 失误中快速学习：Cushman & Wakefield 的真实破局之道 Collin Campbell 认为，衡量企业 AI 成熟度的真正标准在于组织从不可避免的失败中恢复的速度，而非表面的采用率曲线。结合 Cushman & Wakefield 的实际失误（例如过时 PowerPoint 课件导致的聊天机器人数据污染，以及缺乏依据的智能体预测），他总结了三种主要的失败模式：“幽灵效应”（the Ghost Effect）、“自信捏造”（Confident Fabrication）和“试点蜂群”（Pilot Swarms）。为应对这些挑战，该公司推行了一套八问准入框架、严格的 90 天试点阶段关卡，以及“捕获、追踪、修复与复利积累”（Catch, Trace, Fix, and Compound）的迭代学习闭环。该框架推动企业从被动猜测转向系统化的制度性记忆，同时落实了严格的数据所有权归属、强制审计追踪以及硬性的 ROI 考核标准。 Cushman & Wakefield 业务信息官 Collin Campbell 指出，衡量企业 AI 成熟度的关键并非员工采用率或席位数，而是组织从运行故障中系统化恢复并沉淀经验的速度。 (1:56) 企业在落地 AI 时往往面临三大失误模式：非结构化遗留数据干扰检索的“幽灵效应”（Ghost Effect）、缺乏验证的 Agent 幻觉（Confident Fabrication），以及各部门无序发起的“试点蜂群”（Pilot Swarm）。 (5:09) 针对涉及量化预测的 AI Agent，团队确立了“无法证明运算逻辑则严禁上线”的铁律，必须为所有数值输出提供确定性的计算审计轨迹与人工复核，杜绝虚假的“统计模拟”数据。 (12:09) 组织推行“集中护栏、放权想象力”的治理框架，设置最长 90 天的试点生命周期，并通过“捕获、溯源、修复、复利”（Catch, Trace, Fix, Compound）闭环将失误转化为系统级架构规范与 Prompt 规则。 (15:29)

Collin Campbell, Chief Information Officer of Recurring Revenue at Cushman & Wakefield, argues that enterprise AI maturity is measured not by user adoption curves, but by how quickly and systematically an organization recovers from operational failures. Recovery Time Over Adoption Metrics (1:56) Cushman & Wakefield operates globally across commercial real estate with roughly 65,000 employees under recurring, three- to seven-year service contracts. Pointing to industry estimates that 88% of organizations use AI while MIT finds 95% of enterprise generative AI pilots fail, Campbell contends that distributing LLM seats produces only superficial gains. Mature engineering requires instrumenting systems to capture failures and translate every misstep into automated preventative governance. To codify this feedback cycle, the firm applies a four-part loop: Catch, Trace, Fix, and Compound. The final phase embeds root-cause remedies directly into the baseline engineering environment—such as system prompts and coding tools—ensuring previous operational failures cannot recur. By replacing ad hoc troubleshooting with institutionalized architectural rules, Cushman & Wakefield reduced its ideation-to-production deployment cycle from two to three weeks down to roughly two days. Unstructured Data Governance and Data Ownership (7:22) Early deployments revealed critical blind spots in unstructured repositories such as SharePoint, where legacy slides and outdated documents linger as retrieval hazards. During an internal demonstration, an enterprise chatbot identified the head of technology using a 2016–2017 PowerPoint deck rather than the current leadership. The failure illustrated that ingestion pipelines require explicit data boundaries rather than indiscriminate indexing. In response, the firm established that enterprise knowledge retention is a mandatory business deliverable rather than a routine compliance exercise. Cushman & Wakefield introduced clear data domain authority through designated Global Process Owners, instituted strict query filters such as a two-year recency cutoff, and partitioned query boundaries to eliminate the risk of cross-client data contamination. Mandatory Traceability and Validated Math (12:09) A recurring failure pattern emerged in automated agent sandboxes that lacked granular auditability. In one instance, an experimental revenue forecasting agent produced plausible figures that appeared correct on the surface, but deeper inspection revealed the model had bypassed authoritative data sets to generate a 'statistically simulated' hallucination. This breakdown resulted in an enterprise principle prohibiting the deployment of any numerical application that cannot programmatically prove its outputs. Financial workflows now require mandatory, human-in-the-loop checks and step-by-step mathematical long division before any calculated metrics are shared with end users or incorporated into operational forecasts. Centralized Governance and Pilot Stage Gates (15:29) To rein in an uncoordinated influx of rogue field experiments and vendor pitches, Cushman & Wakefield centralized guardrails while decentralizing idea generation. The organization instituted a mandatory request registry and established a strict 90-day maximum stage gate for pilots. This hard time cap prevents open-ended proofs-of-concept from hardening into unvetted production workflows, while granting complex extraction models—such as lease abstraction engines—sufficient repetition across thousands of documents to achieve reliable accuracy. The review process requires answers to eight non-technical intake questions evaluating data grounding, stake-holder visibility, ownership, and failure response. Crucially, proposed agents must detail explicit kill switches, outlining who monitors the agent, how an immediate operational shutdown is triggered, how data is unwound, and what downstream systems are impacted. Executive Buy-In and Strategic Trade-Offs (34:19) C-suite commitment was achieved only after IT shifted its messaging away from optimistic upside presentations toward rigorous threat modeling, detailing how generative models could displace key commercial real estate workflows. Operationally, the firm enforces hard ROI standards—requiring evidence of substantial returns before funding development—while monitoring expense reports and internal API request pipelines to detect unsanctioned shadow AI applications. Reflecting on the rollout, Campbell acknowledges that centralized governance remains an intentional bottleneck because enterprise risk, information security, and brand reputation require human oversight. He also notes two major strategic regrets: delaying a necessary overhaul of IT leadership talent, and temporarily being distracted by standalone, flashy vendor point solutions rather than investing solely in scalable, programmatic architecture. Cushman & Wakefield's AI framework prioritizes rapid recovery and strict auditability over headline adoption statistics, yet Campbell concedes that the strategy and underlying data assets remain an evolving work in progress.

Cushman & Wakefield 持续性收入业务首席信息官 Collin Campbell 认为，衡量企业 AI 成熟度的关键并非技术采纳率，而是系统从运行故障中恢复和学习的速度。企业必须建立以严格数据确权、输出溯源以及中央护栏为核心的治理体系，才能将生成式 AI 试点转化为可持续的商业价值。 重构 AI 成熟度：从采纳曲线转向恢复速度 (1:56) 当前企业界普遍将 AI 实施聚焦于泛化采纳，例如为全员配发 Claude、ChatGPT 或 Gemini 账号。然而根据 McKinsey 和 MIT 的相关统计，尽管绝大多数企业已开始使用 AI，但仅有 34% 在主动重构业务流程，且高达 95% 的企业生成式 AI 项目最终走向失败。Campbell 指出，单纯分发模型许可证只能带来表层生产力提升，无法产生结构性价值。真正的企业 AI 成熟度取决于“恢复时间”，即团队在遭遇不可避免的技术失误后，能否迅速定位根因，并将其沉淀为预防性的治理系统，而非简单抛弃失败试点并盲目转向下一个概念验证。 企业落地的三大暗坑：幽灵数据、自信捏造与试点泛滥 (5:09) Campbell 归纳了企业级 AI 最常遭遇的三大失误模式。第一是“幽灵效应”（The Ghost Effect），传统数据治理长期偏向 SQL 等结构化数据库，往往忽视了 SharePoint 等散乱的非结构化知识库；Cushman & Wakefield 的内部测试曾因此遭遇尴尬，其聊天机器人根据 2016 至 2017 年的一份陈旧 PPT，误报了早已卸任的高管姓名。第二是“自信捏造”（Confident Fabrication），早期缺乏追踪机制的 Agent 曾将完全虚构的财务预测粉饰为“统计模拟输出”。第三是“试点蜂群”（The Pilot Swarm），各业务单元与第三方供应商自发推进零散的单点试验，缺乏集中评估与长远价值。 治理双重铁律：明确数据责任与强制过程推理 (10:46) 为破除上述陷阱，企业确立了两项底层研发原则。首先，知识沉淀不再是一项合规任务，而是核心业务交付物。Cushman & Wakefield 设立了全球流程所有者（GPO），明确核心数据领域的归属权（例如由人力资源部门对组织架构数据全权负责），并在检索增强生成（RAG）管道中强制执行严格的两年内数据时效过滤，同时严格杜绝不同租户间的数据交叉污染。其次，面对数值计算与业务推理，系统必须遵循“展示解题步骤”的原则；若无法以可量化的事实依据与程序化方式自证其输出逻辑，无论交互界面多么精美或声称的效率增益有多高，一律不得发布上线。 中央护栏与业务闭环：90天试水期与八项准入质询 (15:29) 在运营管理层面，公司推行“集中护栏，放开想象力”的协同模式。所有 AI 创意必须统一录入需求登记库，并设定最长 90 天的阶段性关卡。Campbell 强调，长达半年的试点已失去试验意义，而复杂任务（例如商业租赁合同要素抽取）则需要充足的实操样本迭代才能达到工业级精度。通过推行由“捕获（Catch）、溯源（Trace）、修复（Fix）、沉淀（Compound）”构成的学习闭环，团队将研发周期从数周压缩至约两天。此外，针对每个拟建项目，必须经过包含“若项目失败谁会察觉”以及“如何立即执行终止与数据回滚”等八项务实问题的准入评审。 高管对齐与架构复盘：从节流叙事转向防范性增长 (33:48) 在推动组织文化变革时，泛泛描绘 AI 发展潜力的汇报往往难以打动 C-suite 高管；只有通过严谨的威胁建模，清晰推演 AI 对商业地产模式可能造成的颠覆性冲击，才能真正促使最高管理层下定治理决心。在复盘既往决策时，Campbell 坦言其最大遗憾在于过晚启动人才梯队重构，因为传统 IT 领导力的技能模型无法无缝迁移至生成式工作流；同时，在外部推销压力下采购特定单点应用，也曾一度分散了构建可持续系统架构的精力。企业当前更强调硬性投资回报（如 10 倍收益标准）与长效架构，逐步摆脱以裁员或单纯节省工时为导向的窄化叙事。 Cushman & Wakefield 的实践表明，企业推行生成式 AI 的重心应从统计使用率转向建立自愈型组织记忆。尽管集中化的治理模式难免带来一定审核摩擦，但通过将每次失败制度化地转化为系统规则，企业才能在防范信息安全与品牌风险的同时实现规模化扩张。

## Background 本视频由 Ai4 于 2026 年 8 月 19 日在 YouTube 平台发布。内容为戴德梁行（Cushman & Wakefield）经常性收入业务首席信息官 Collin Campbell 的主题演讲，深入分享了企业级 AI 落地过程中因脏数据、错误假设与试点失控所遭遇的真实挫折，以及如何将这些失误转化为治理、采纳与快速交付的实战经验。 ## TL;DR Collin Campbell 指出，企业 AI 成熟度的关键衡量标准是系统的故障恢复力与认知迭代速度，而非单纯的工具使用率。戴德梁行在经历“幽灵效应”（索引陈旧数据）、“自信胡说”（模型伪造推导逻辑）和“试点蜂群”（分散无序立项）等阵痛后，确立了“知识保留即业务交付”与“无法自证便不上线”等第一原则。团队通过实施最长 90 天的阶段关卡淘汰制、单一责任人制度以及“捕捉-溯源-修复-复利”的闭环学习机制，将每一次错误转化为注入开发工具链的系统护栏，成功将 AI 项目从构想到上线的交付周期压缩至约 2 天，实现了以硬性 ROI 和业务增长为核心的规模化转型。 ## Atmosphere & style 整场演讲风格极其务实、坦诚且充满幽默感，主讲人毫不避讳地公开自嘲并复盘团队内部的各种翻车案例。现场节奏明快生动，既有将模型训练比作“健身房举铁做组”等接地气的趣味隐喻，又贯穿着对传统 IT 采购模式、虚假繁荣指标的深刻反思，给听众带来如同亲临企业一线复盘复盘会般的真实感与启发感。 ## Key takeaways - [02:31](https://www.youtube.com/watch?v=pucUyA42ZbQ&t=151s) 企业 AI 的成熟度衡量标准不应是员工采纳率或工具发放量，而是系统在出现故障后的恢复时间与防范体系。 - [05:09](https://www.youtube.com/watch?v=pucUyA42ZbQ&t=309s) 企业落地 AI 最易踩中三大陷阱：非结构化历史数据导致的“幽灵效应”、缺乏溯源的“自信胡说”，以及部门各自为政的“试点蜂群”。 - [10:46](https://www.youtube.com/watch?v=pucUyA42ZbQ&t=646s) 第一原则：知识保留与数据生命周期管理不再是应付检查的合规任务，而是直接决定 AI 成败的核心业务交付物。 - [12:09](https://www.youtube.com/watch?v=pucUyA42ZbQ&t=729s) 第二原则：如果 AI 或智能体无法证明自身有效性或展示严谨的推演计算逻辑，一律禁止交付上线。 - [15:29](https://www.youtube.com/watch?v=pucUyA42ZbQ&t=929s) 推行需求登记册与 90 天阶段关卡淘汰制，通过清晰的授权审批机制（DOA）终结无序试点的资源内耗。 - [18:23](https://www.youtube.com/watch?v=pucUyA42ZbQ&t=1103s) 坚持“规则集中化，想象力去中心化”，并推动企业 IT 从纯 SaaS 采购转向自研与外购并行的混合交付模式。 - [19:55](https://www.youtube.com/watch?v=pucUyA42ZbQ&t=1195s) 建立“捕捉、溯源、修复、复利”的学习闭环，将踩坑教训直接注入开发工具链与提示词工程，形成复利护栏。 - [22:03](https://www.youtube.com/watch?v=pucUyA42ZbQ&t=1323s) 推行试点单一负责人制并强制事故复盘挂钩系统变更，将项目从构想到上线的周期从 2-3 周缩短至 2 天。 - [24:06](https://www.youtube.com/watch?v=pucUyA42ZbQ&t=1446s) 在需求准入阶段引入熔断机制（Kill Switch），确保在异常失控时有明确的责任人能够即刻叫停并回滚操作。 - [26:51](https://www.youtube.com/watch?v=pucUyA42ZbQ&t=1611s) 停止单纯考核许可证或 Token 消耗等表面使用率，关注认知迭代速度，推动试点项目“毕业”为成熟的企业数字资产。 - [31:11](https://www.youtube.com/watch?v=pucUyA42ZbQ&t=1871s) 90 天评估上限是为模型在复杂垂直场景（如租约抽象）中提供充分的数据磨炼与验证周期（Reps）。 - [34:19](https://www.youtube.com/watch?v=pucUyA42ZbQ&t=2059s) 全公司应建立“硬性 ROI”衡量共识，并利用解决首个问题的经验前瞻性推导出后续防护规则。 - [38:23](https://www.youtube.com/watch?v=pucUyA42ZbQ&t=2303s) 企业推进 AI 变革应及早布局人才战略，避免盲目追逐光鲜亮丽的外部演示工具，专注打磨底层演进机制。 ## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=pucUyA42ZbQ&t=4s) 开场与背景：实用主义与高韧性的企业 AI 战略 在技术峰会第三天的演讲中，Collin Campbell 指出，行业内充斥着大量关于技术架构与概念验证的宏大叙事，但他更希望带来一场毫无保留、坦诚务实的真实经验分享。虽然部分内容为了遵守市场与公关（PR）要求进行了脱敏处理，但其核心仍然聚焦于企业在真实部署中遭遇的挫折、从中吸取的教训以及后续采取的差异化应对手段。 Collin Campbell 目前担任高力国际/戴德梁行（Cushman & Wakefield）经常性收入（Recurring Revenue）业务的首席信息官（CIO/BIO）。Cushman & Wakefield 是一家拥有 65,000 名员工的全球性房地产服务巨头，其经常性收入部门涵盖楼宇运营、物业资产管理以及投资组合估值分析等核心业务。这类业务通常签署为期 3 至 7 年的长期服务协议，要求 IT 团队每一天都必须向客户证明持续交付的增量价值。“炫酷的演示（splashy demo）毫无用处，真正重要的是产生实际投资回报率（ROI）并为客户带来实质效益。”Collin 强调，AI 在本质定义上往往带有脱离实际的探索性，因此技术领导者的核心任务就是为其注入严谨的商业实用性。 - **个人管理哲学**：极度务实与价值驱动（Pragmatic & Value-driven），拒绝只看不中用的技术噱头，专注于削减成本、交付增量价值与构建差异化壁垒。 - **个人趣味**：热衷于主题乐园、过山车，并且是万圣节装扮的忠实爱好者。 - **演讲议程**：通过三大真实故事剖析企业 AI 的常见模式与根因，并推导出相应的治理框架与实操手册（Playbook）。 --- ### [02:31](https://www.youtube.com/watch?v=pucUyA42ZbQ&t=151s) AI 成熟度的新定义：从“采纳率”转向“系统恢复力” 演讲引用了一组发人深省的行业统计数据来切入当前 AI 落地的真实困境：麦肯锡（McKinsey）数据显示 88% 的企业正在日常使用 AI（Collin 认为目前该数字已接近 98%），其中 34% 的企业正在围绕 AI 重构工作流程；而麻省理工学院（MIT）此前的报告指出，高达 95% 的企业级生成式 AI（Generative AI）项目最终以失败告终。这引发了一个深层次问题：企业到底是在真正重塑业务，还是仅仅在追求微小的生产力改善？ 针对企业 AI 能力的评估，Collin 提出了一个反传统的判断标准：“AI 的成熟度曲线衡量的不应是采纳率（Adoption），而是系统在故障后的恢复时间（Recovery Time）。”如果直接给全公司 98% 的员工发放 Claude、OpenAI ChatGPT 或 Google Gemini 的授权许可，员工的个人工作效率或许会提升，但这并不意味着企业自动获得了商业价值。 ```mermaid flowchart TD accTitle: AI Failure and Recovery Framework accDescription: 传统工程方法与 AI 实验迭代在面对错误时的机制对比 subgraph Trap["常见误区：无节制试错"] A["发生错误 (Oops)"] --> B["放弃深入复盘"] B --> C["快速跳向下一个用例"] end subgraph Framework["成熟度核心：恢复力与工程化体系"] D["发生错误 (Oops)"] --> E["基础度量与根因剖析"] E --> F["构建防范体系与治理机制"] F --> G["系统性拦截未来同类故障"] end ``` 由于 AI 工具的构建速度极快，团队往往缺乏学习失败的耐心，容易在用例失败后立即跳向下一个热点，导致问题在底层反复发生。企业必须引入类似于传统软件工程的度量与监控机制，将每次实验中的“翻车（Oops）”转化为可防范同类错误重复发生的体系化规则。 --- ### [05:09](https://www.youtube.com/watch?v=pucUyA42ZbQ&t=309s) 企业 AI 三大典型陷阱：幽灵效应、自信胡说与试点蜂群 基于 Cushman & Wakefield 的实际试错经历，Collin 总结了企业在智能化转型早期极易踩中的三大根因模式： 1. **幽灵效应（The Ghost Effect）**：遗留的历史废弃数据、常年未被下架的陈旧 PDF 以及过时的组织架构图潜伏在企业存储系统中。传统数据治理通常只关注规范化表格数据库（如 SQL、Salesforce 或 Oracle），却完全忽视了如 SharePoint 等非结构化文档库的混乱现状。 2. **自信胡说（The Confident Fabrication）**：在早期探索智能体（Agentic）与沙箱环境时，由于缺乏足够的可追溯性，模型经常生成看似极具说服力却完全错误的答案。当时团队得到的解释令人哭笑不得——“模型并非在产生幻觉（hallucinating），而是在进行统计学上的改进与即兴发挥（statistically improving and improvising）”。 3. **试点蜂群（The Pilot Swarm）**：去中心化的业务部门自发构建零散工具，加之外部供应商推销各类包装方案，导致企业内部充斥着各式各样的 AI 试点，形成无序的“百家宴（potluck meal）”，既无法统一管控，也未能真正驱动核心业务指标前行。 --- ### [07:22](https://www.youtube.com/watch?v=pucUyA42ZbQ&t=442s) 实战案例一：被时间遗忘的聊天机器人与数据治理之痛 作为企业最常见的首发场景之一，Cushman & Wakefield 早期推出了一款内部知识问答聊天机器人，旨在标准化解答员工日常高频问题（如假期日历查询、业务对接人检索等）。系统在进入第二至第三阶段的推广测试时，整体运行表现令人满意。 然而，在一次面向技术全员的大型演示中，有员工向该机器人提问了一个基础组织问题：“当前负责技术最高决策的执行高管（Executive in charge of technology）是谁？”该聊天机器人却非常自信地给出了现任主管前任的名字。当团队追溯其检索依据时，发现机器人调出了一份保存在非结构化知识库中、2016 至 2017 年间的旧 PowerPoint 演示文稿。 ```mermaid flowchart LR accTitle: Ghost Effect in Enterprise Search accDescription: 聊天机器人因索引过期文档导致回答错误的流程示意 subgraph Storage["企业存储库"] DB["结构化数据库 (SQL / Salesforce)"] SP1["现行业务文档"] SP2["2016年旧 PPT (未被下架)"] end subgraph Chatbot["AI 检索体系"] Query["用户提问：技术负责人是谁？"] --> RAG["未设时效过滤的检索"] SP2 --> RAG RAG --> WrongAns["错误输出：前任技术高管"] end ``` 这一尴尬但颇具代表性的事件暴露出企业在数据摄取与数据权属治理上的巨大漏洞： - **数据摄取边界**：并非所有历史文档都有价值，必须在摄取前对非结构化文档库进行严格的生命周期管理。 - **数据所有权模糊**：究竟由全球流程负责人（Global Process Owners）还是具体领域负责人（Domain Owners）来确认组织实体的“单一事实来源（Single Source of Truth）”。 - **企业级推诿**：发现脏数据后，修复责任往往难以落实，宛如家长让孩子打扫房间时孩子反呛“这不是我的工作”。 - **治理举措落地**：团队随即对知识检索系统制定了强约束规则，引入明确的“数据时效性窗口（Recency Rule）”（例如默认仅索引两年内的数据）。 ### [10:46](https://www.youtube.com/watch?v=pucUyA42ZbQ&t=646s) 核心数据域权属与第一原则：知识保留即业务交付 在划定核心数据域时，最关键的问题是明确谁才是数据的真正负责人。以组织架构图为例，其合规且权威的所有权显然属于人力资源或人才团队（People Team）。如果无法界定权属并把控数据的时效性与准确性，一旦进入直面客户的真实商业场景，就会引发荒谬且危险的事故——比如在检索百事可乐（Pepsi）的客户资料时，系统却混入了可口可乐（Coca-Cola）的数据。为此，团队开始紧密围绕数据所属领域、数据生命周期与新鲜度，以及“谁有权决定最终正确答案”来重构企业数据体系。 由此，戴德梁行（Cushman & Wakefield）确立了其 AI 战略的第一项核心原则：“知识保留不再是一项合规任务，而是一项业务交付物”（"Knowledge retention isn't a compliance task anymore"）。传统观念中人们习惯于使用数据治理（Data Governance）、数据管理（Data Stewardship）或数据清洗等中规中矩的合规术语；但在 AI 时代，核心焦点已经从单纯的“数据”升级为“企业知识”。确保这些核心知识在系统中沉淀与留存，是直接驱动业务价值的战略交付，而非应付检查的合规流程。 - **核心数据域梳理维度**：所属业务领域（Domain）、数据产生年限（Age）、数据时效性（Recency）、权威答案的控制与仲裁权（Authority）。 - **第一原则（Principle 1）**：知识保留从“合规任务”转变为“核心业务交付物”。 --- ### [12:09](https://www.youtube.com/watch?v=pucUyA42ZbQ&t=729s) “打破数学常理”的预测失误与第二原则：无法自证便不上线 这一阶段的一起典型翻车事故源于一次内部实验。在 AI 应用早期，团队尝试构建自动化智能体（Agent）来执行基础的收入预测，原本期望通过输入历史趋势、市场因素和商机漏斗（Pipeline）数据，由智能体自动推演未来 3 个月、6 个月和 12 个月的指标，以彻底取代传统繁复的 Excel 手工测算。然而，智能体最终给出的预测数字乍看之下非常合理，但由于当时并未对智能体施加“解释推演过程”的约束，团队随后要求它“像做长除法一样展示计算步骤并证明其数学逻辑”。 结果令人哭笑不得：智能体给出的计算逻辑完全是“统计学模拟（Statistically Simulated）”出来的。换言之，模型根本不知道自己是如何算出这一结果的，它只是编造了一套看起来像模像样的幻觉谎言，完全无视了后台那些经过精心治理、权属分明的高质量底层数据。幸运的是，该问题在沙箱环境（Sandbox）中被及时拦截。团队随即调整了验证机制：要求智能体的所有输出指标必须带有可量化的计算链路，并在引入全自动数据认证前强制执行人工交叉核验。 这起失误催生了第二项核心开发原则：“如果它无法向你证明自身有效或展示工作逻辑，就绝不上线”（"If it can't prove to you that it works or shows it works, don't ship it"）。无论前端界面多么酷炫、号称能节省多少生产力工时，只要系统无法确凿证明“1 + 1 = 2”的严谨性，就绝对不能推向生产环境。 - **失误根源**：未要求智能体展示可溯源的计算逻辑，导致模型依赖统计幻觉编造预测值。 - **修复方案**：沙箱拦截、强制输出量化验证链条、引入人工校准与数据认证机制。 - **第二原则（Principle 2）**：无法提供可复核自证逻辑的 AI 应用，一律禁止交付上线。 --- ### [15:29](https://www.youtube.com/watch?v=pucUyA42ZbQ&t=929s) 纸飞机般的试点洪流与 90 天阶段关卡制治理 伴随 AI 热潮的兴起，业务部门提出了海量的创新构想，各种项目提案如同“纸飞机”一样不断抛向技术团队。这些提案描绘了各种将商业地产业务全流程数字化的宏大构想，但实际落地时却暴露出重重隐患：变革管理（Change Management）与用户采纳率极低、缺乏企业级扩展能力、实际功能与宣传严重不符。由于初期缺乏集中的合规框架、护栏约束、可衡量的业务影响追踪机制以及统一的 AI 资产组合管理（Portfolio Ownership），导致企业内部充斥着巨大的协同噪音与资源浪费。 为了终结这种混乱，团队推行了一套结构化的治理机制：首先建立统一的“需求登记册（Request Registry）”；其次对试点项目设立严格的阶段关卡（Stage-Gate），规定所有试点的周期上限为 90 天——任何拖延至 3 到 6 个月的试点都不再是验证，而是演变成了低效的常态工作；最后推行强制决策机制（Forced Decision-Making），明确界定各层级主管的授权权限（Delegation of Authority, DOA），明确谁有权在区域、业务线乃至全公司范围内做出“继续推进或直接叫停”的决定。这套流程不仅大幅强化了业务端的采纳意愿，更有效遏制了重复造轮子的资源内耗。 ```mermaid flowchart TD accTitle: AI试点治理阶段关卡流程 subgraph INTAKE["1. 需求准入"] A["业务创新构想"] --> B["需求登记册<br>(Request Registry)"] end subgraph STAGE_GATE["2. 90天关卡制验证"] B --> C["阶段关卡验证<br>(Stage-Gate Pilot)"] C --> D{"试点周期 <= 90天?"} D -- "超时" --> E["强制关停 / 清理退出"] D -- "合规" --> F["量化成效与扩展性评估"] end subgraph DECISION["3. 决策与投产"] F --> G{"DOA 明确授权决策<br>(Delegation of Authority)"} G -- "否决" --> E G -- "批准" --> H["规模化投产上线<br>(避免重复建设)"] end ``` - **试点治理痛点**：缺乏组合管理，多部门重复开发类似工具（如多个团队分别开发应付账款核算或工程调度工具）。 - **四步治理措施**： 1. 建立需求登记册（Request Registry）。 2. 实施最长不超过 90 天的阶段关卡淘汰制。 3. 落实清晰的授权审批机制（DOA），压实决策责任。 4. 制定严密的生产级扩展标准，保障业务协同与投产成效。 --- ### [18:23](https://www.youtube.com/watch?v=pucUyA42ZbQ&t=1103s) “规则集中化，想象力去中心化”与混合交付模式 在总结试点治理经验时，技术团队提炼出了核心协作准则：“集中护栏，去中心化想象力”（"Centralize the guardrails, decentralize the imagination"）。技术领导层不应自负地认为自己掌握了房地产科技创新的全部答案，而应当广泛汲取业务一线的创新洞察，同时在中心端制定统一、严密的安全与合规护栏，确保每一次落地交付都能兼顾敏捷性、安全性与确定性商业价值。 与此同时，企业的 IT 交付模式也发生了根本性变革。戴德梁行以往是一家几乎 100% 依赖采购成熟 SaaS 软件的“采购型组织（Buy Shop）”，过去从不涉足自主代码开发。而现在，团队全面转型为“自研与外购并行的混合模式（Hybrid Model）”。将决策框架嵌入护栏后，团队能够清晰裁决何时通过采购特定外部系统以实现极速赋能，何时依托内部工程自研开发，从而在支持去中心化业务创新的同时，实现对庞大商业地产业务生态的高效支撑。 - **协作理念**：中心端把控安全合规标准，业务端释放创新应用场景。 - **交付模式转型**：由纯 SaaS 软件采购转向“自研 + 外购”的混合架构模式。 --- ### [19:55](https://www.youtube.com/watch?v=pucUyA42ZbQ&t=1195s) 构建 AI 学习闭环：从踩坑中提炼复利护栏 为了让技术组织以更低的代价加速演进，团队建立了一套旨在“加速遭遇失败并提炼经验”的学习闭环（Learning Loop）。正是因为早期踩过了上述各种陷阱，团队如今不再需要担忧模型调用 6 年前的陈旧数据，也不用反复排查数据权属问题，而是能够直接进入“数据认证机制”等更高级的命题。整个闭环包含四个不可或缺的阶段： ```mermaid flowchart LR accTitle: AI 学习闭环四步法 A["1. 捕捉 (Catch)<br>工具与流程侦测异常"] --> B["2. 溯源 (Trace)<br>深究根因而非表面现象"] B --> C["3. 修复 (Fix)<br>改写代码 / 删除陈旧资产"] C --> D["4. 复利 (Compound)<br>将护栏注入开发工具链"] D --> A ``` 该闭环最核心的秘诀在于第四步“复利（Compound）”。通过将修复经验提炼为系统护栏，直接注入日常使用的工程工具链中——无论是配置在 Claude Code、Cursor 等代码辅助工具内，还是固化到基础提示词工程或 ChatGPT 技能（Skills）中——让所有后续开发自动引用最新的架构标准与治理规则，从而使每一次历史教训都转化为推动未来开发提速的确定性资产。 - **学习闭环四步法（Learning Loop）**： 1. **捕捉（Catch）**：建立完备的工具链与监控流程，主动捕捉 AI 犯下的细微错误。 2. **溯源（Trace）**：深挖根本原因（例如精准识别是“缺少文档保留策略”还是“偶发调取了旧文档”）。 3. **修复（Fix）**：通过调整底层代码、清理过期文档或关闭安全漏洞快速止损。 4. **复利（Compound）**：将治理规则沉淀为系统级护栏，深度嵌入 Claude Code、Cursor、基础 Prompt 模板及 GPT Skills 中，实现组织经验的代际继承。 ### [22:03](https://www.youtube.com/watch?v=pucUyA42ZbQ&t=1323s) 从盲目试错到系统化学习：缩短交付周期的进化之道 在 Cushman & Wakefield 早期推进 AI 项目时，一个想法从萌芽到最终上线，平均需要耗费两到三周的时间。团队不得不反复进行审查、验证与修补。然而，通过持续调整机制，如今从创意构想到投入生产环境的时间已经被压缩至短短两天。Collin Campbell 坦言团队并不完美，但他深入对比了“盲猜型企业”与“学习型企业”在处理 AI 上的本质差异：盲猜型企业四处跟风做 Pilot（试点项目），只是为了对外展示存在感或抢占市场噱头，结果往往是在同一个坑里跌倒多次，甚至在开会时总喜欢敷衍地说一句：“我们就先保持关注吧，继续监控它，反正出了一次错又不一定会出第二次、第四次。” 与此相对，“学习型企业”则建立了一套严密的反思链条。任何故障都能直接追溯到底层数据源——无论是具体的文档、数据库还是边缘业务系统。此外，公司推行了清晰的单一负责人制度，必须有具体的负责人站出来宣称：“这是我的点子，我全力看好它，我们要把它推落地。”更关键的是，团队将每一次失误都转化为下一次迭代的护栏，先是在敏捷冲刺（Sprint）规划中吸取教训，随后通过自研的准入（Intake）工具将这些经验固化为系统性规则，并强制推行事后复盘（Post-mortem），确保每次事故都能直接促成系统架构或业务流程的实质改进。 ```mermaid flowchart TD accTitle: AI 试点落地与反馈闭环 subgraph Intake["需求准入与立项"] A["业务提出想法"] --> B["自研准入 App: 8 大关键质询"] B --> C{"单一负责人认领并评估可行性"} end subgraph Delivery["敏捷交付与验证"] C --> D["快速投产 (2-4 天极速交付)"] D --> E["业务磨炼与积累 Reps (上限 90 天)"] end subgraph Feedback["复盘与防护机制"] E --> F["故障触发或异常监控"] F --> G["根因追溯 (数据/文档/底层接口)"] G --> H["沉淀为系统护栏与流程规则"] H -.-> B end ``` - **交付周期跃迁**：AI 项目从创意到生产环境的部署周期从原先的 2 至 3 周大幅缩短至约 2 天。 - **项目归属原则**：每个试点项目必须配备单一负责人（Owner），彻底杜绝无主项目和无人问责的实验。 - **闭环复盘机制**：事故复盘必须直接挂钩系统变更（System change）或流程改造（Process change），严禁口头承诺式的“持续监控”。 --- ### [24:06](https://www.youtube.com/watch?v=pucUyA42ZbQ&t=1446s) 需求准入的八大关键质询与熔断机制 为了避免重蹈覆辙，Cushman & Wakefield 将过去 18 个月摸爬滚打得出的血泪教训，凝练成自研需求准入应用中的八个核心问题。这些问题刻意剥离了晦涩的技术黑话，让业务人员在填写申请时能够清晰理解并作答。Collin 重点剖析了其中几项最具实战价值的考量：第一是数据基础，“这到底是有真实数据作为锚定，还是纯粹凭感觉？”在做市场创意时或许可以依赖灵感，但如果要分析诸如纽约市租金等严谨指标，就必须扎根于客观事实；第二是能见度（Visibility），“我们究竟什么时候才真正关注这些数据？这是全新的业务还是现有流程的一部分？” 第三点则是 Collin 个人最推崇的风险假想：“如果我们搞砸了，谁会第一时间发现？”这种预警涵盖内部系统的红灯报警机制，更关乎面向一线现场或外部客户时的响应敏感度。紧随其后的考量是治理纪律与“熔断开关”（Kill Switch）的设计。企业在谈论 Agent 时常常堆砌各种高大上的术语，但最根本的问题在于：到底有没有人在盯着这个跑来跑去的代理？一旦情况失控，如何立刻叫停？怎样评估连锁影响？又该如何将操作优雅地回滚？只有当业务方能清晰回答这些逻辑，并能在系统层面实施干预时，项目才被允许启动。 --- ### [26:51](https://www.youtube.com/watch?v=pucUyA42ZbQ&t=1611s) 终极心法：不看使用率，让试点项目真正“毕业” 面对技术团队与业务伙伴，Collin 树立了一条核心管理准则：“我们不试图阻止错误的发生，我们追求的是从错误中快速学习，确保未来绝不再犯。”这种快速试错与复盘的透明过程不仅沉淀为团队的知识库，更转化为敏捷的“肌肉记忆”，并在技术与业务端之间建立起珍贵的信任纽带。在演讲临近尾声时，他给出了三条颠覆传统思维的重磅建议： 首先是**永远不要隐瞒失败**。Collin 调侃道，自己绝不会把写着“AI 大败局”的幻灯片印成圣诞贺卡寄给亲友，但既然敢于站在大会舞台上公开自嘲与剖析，任何人就都应当有勇气正视失败，把挫折淬炼成加速前行的燃料。其次，**彻底停止考核使用率（Utilization）**。许多组织热衷于炫耀发放了多少张软件许可证、消耗了多少 Token，然而 Collin 明确表示：“我不再衡量使用率了，去衡量你们学得够不够快、创造了什么实质价值。”哪怕一次尝试以失败告终，只要带来认知跃升并杜绝再次犯错，这就是核心价值所在。 最后，**不要只做试点规模扩张，而要让试点项目“毕业”**。很多团队津津乐道于将试点用户从 5 个人扩大到 50 人、100 人，但这只是数字游戏；真正的成功是将这些验证过的能力转化为具有制度化规模的成熟产品，让它作为企业的“数字资产”（Digital Asset）在整体 IT 生态中自主生长与反哺。 > “洞察的速度远胜于上线的速度（Speed to insight beats speed to launch）……如果 AI 无法解释自己的推演逻辑，如果智能体说不清自己在干什么，你不过是在侥幸碰运气，它根本还不成熟。” - **学习驱动替代指标浮夸**：摒弃单纯统计 License 派发量或 Token 吞吐量的虚假繁荣，将“认知迭代速度”作为首要度量衡。 - **让试点正式毕业**：跳出小圈子用户数字的横向扩展，注重沉淀为活在企业架构内部的常态化数字资产。 - **可解释性底线**：无法自我解释决策路径的 Agent 属于不成熟方案，必须设立刚性护栏以确保服务质量。 --- ### [31:11](https://www.youtube.com/watch?v=pucUyA42ZbQ&t=1871s) 现场问答：90 天上限、FOMO 驱动与影子 AI 排查 在问答环节，现场观众就部署周期的矛盾点率先发问：既然此前提到部署仅需 2 到 4 天，为何评估框架中又设立了长达 90 天的最大关卡（Gate）？Collin 借用健身房举铁的隐喻回答道：“有时候你需要做组（Reps）。”以商业地产中的租约抽象（Lease Abstraction，即利用 AI 进行租约合同数字化解读）为例，AI 处理前 10 份合同可能表现平平，批处理 2000 份时有了起色，直到吃透 10000 份合同后才表现卓越。这一过程既是对模型能力的打磨，也是对业务理解的反复淬炼；如果机械地在第四天就强行叫停，极可能会错杀那些需要数据反复喂养才能释放潜力的优秀场景，因此 90 天是留给大模型做足“重复训练与验证”的保护上限。 针对现场关于如何推动企业文化转型的提问，Collin 引发了全场阵阵笑声。他坦言这场变革至今仍在进行，不仅包含自上而下的推动与自下而上的探索，其中最强大的助推催化剂其实是 **FOMO（错失恐惧症）**——当员工看到身边的同事用 AI 搞出了惊艳的成果，就会坐立不安地惊呼：“天呐，我也得学会这个，这太酷了！”然而这种狂热也曾带来副作用，团队一度从僵化的指令控制模式滑向过于松散的各自为战，导致不同员工交付的产出甚至与公司品牌调性产生脱节。目前团队更倾向于采取“胡萝卜加大棒”中的胡萝卜策略来引导转型。 针对如何摸排企业内部未登记的“影子 AI”（Shadow AI）问题，Collin 分享了极为接地气的实用手段： - **财务开销全面审计**：直接审计各部门的费用报销与采购支出流向，快速锁定员工自费订阅外部 AI 工具的痕迹。 - **数据交互与接口监控**：严密监测底层的数据管道（Pipeline）访问申请、API 调取请求以及向外的数据交换行为，顺藤摸瓜排查出那些处于盲区之中的未知实验。 ### [03:48](https://www.youtube.com/watch?v=pucUyA42ZbQ&t=228s) 建立单一试点漏斗与设定以“增长”为导向的企业级目标 在试点项目和组织变革的推进中，高层管理团队支持建立一个“单一漏斗”（Singular Funnel），以便在整个企业级层面上进行规模化推广。然而，接下来亟待推进的关键变革是为这些项目设定真正的企业级量化目标。与其设定听起来偏向负面缩减的“自动化指标”，不如将衡量标准聚焦于业务增长。企业应当关注的是在特定领域内实际落地执行了多少次核心动作，把重点从单纯的“节约成本”彻底转向“驱动业务增长”。 ```mermaid flowchart LR accTitle: AI项目规模化推进漏斗 subgraph S1["输入与管理"] A["单一试点漏斗"] --> B["高管层战略支持"] end subgraph S2["落地与成效"] B --> C["企业级量化目标"] C --> D["以增长为导向的商业价值"] end ``` ### [34:19](https://www.youtube.com/watch?v=pucUyA42ZbQ&t=2059s) 防护栏建设、硬性 ROI 与 IT 流量管控的瓶颈挑战 现场观众针对管理模式提问，特别提到了“IT 交通管制（IT traffic control）vs TSA 安检”的生动比喻，询问在集中设立防护栏（Guardrails）过程中的实践经验。Collin Campbell 指出，并非每一个防护栏都必须等犯了错才去总结，“要用设立的第一个防护栏，去推导并预判接下来本该提前考虑好的三个防护栏”。在企业文化层面，真正的转折点在于全公司形成了对“硬性投资回报率”（Hard ROI）的共识——比如“周四下午做个简单算术：投了 100 万美元，有没有赚回 1000 万美元？”当这种价值考量成为全公司的共同信念，而非个人偏好时，员工就会找准自身定位，不再试图一人包揽工程师、应用开发、QA 以及最终裁判等所有角色。 与此相对应，IT 扮演“交通警察”依然存在现实瓶颈，因为这不可避免地会让 IT 成为审批与流转的卡点。无论系统流程优化得多顺畅，企业依然必须为品牌声誉、信息安全（Infosec）和变革管理等全方位风险兜底。对此，唯一的解法是通过持续的普及教育和工具采纳，让业务人员清楚理解如何在规范框架下使用企业级工具集。 - **防护栏演进逻辑**：用解决首个问题的经验，前瞻性推导出后续多个关联规则。 - **价值衡量文化**：从“个体随意尝试”转变为“全员坚守硬性 ROI 逻辑”。 - **风险与效率平衡**：在防范品牌、安全和管理风险的同时，依靠普及培训缓解 IT 瓶颈。 ### [36:20](https://www.youtube.com/watch?v=pucUyA42ZbQ&t=2180s) 破解 AI 悖论：从最坏情景建模到分层教育与推进 面对现场关于“AI 充满希望与价值承诺，落地却面临繁重矛盾任务”这一悖论的提问，Collin Campbell 回顾了组织变革管理的探索历程。虽然企业过去一直有各类机器学习应用，但当生成式 AI 带来真正的“灵光一现时刻”时，团队最初对高管层开展了全方位的机遇科普，效果却只能算“差强人意”。随后他们转变策略，对 AI 可能给行业带来的所有极端破坏场景进行建模（包括业务被完全颠覆或人员被全面取代等），这迅速引起了高管们的高度警觉与重视。 在引起高层重视后，策略逐步向下拆解并贯彻到各个业务条线负责人，明确每个人在公司 AI 战略中的具体职责： 1. **基础应用层**：指导员工通过合规受控的企业级渠道安全、高效地使用工具（例如 Claude）。 2. **核心资产层**：针对拥有明确注册负责人的企业级数据资产，严格规范操作权限与处理流程。 ### [38:23](https://www.youtube.com/watch?v=pucUyA42ZbQ&t=2303s) 终极复盘：重构人才战略与拒绝“追逐新奇工具” 在最后环节，观众抛出了一个趣味问题：“有什么是大家还没问、但本来应该问的问题？”Collin Campbell 先是开玩笑说大家应该问他最喜欢的过山车是什么（答案是坦帕市的 Iron Gwazi），随后严肃地给出了他认为最关键的自省议题：“如果现在能回到 2024 年 11 月一切真正开始影响业务的时刻，会有哪两件事选择用完全不同的方式去做？” 第一是**优先启动人才战略**。Collin Campbell 坦言团队在人才布局上起步偏慢，这是一大遗憾。如今成为一名 IT 领导者的要求与过去截然不同，“那些非常优秀、陪伴我走到今天的人，并不一定是带领业务走向未来的人选”。第二是**尽早建立底层系统性防护栏**。面对市场上五花八门的诱人方案——例如应付账款（AP）的惊艳工具、数字化楼宇的差异化方案——人们往往容易被这些“光鲜亮丽的新功能”（shininess）吸引，从而忽视了构建一个具备自学习与持续演进能力的底层机制。如果能回到过去，他会坚定地告诫自己：“别管那个演示有多酷，它对长远发展毫无帮助。” ## Notable quotes > [00:04](https://www.youtube.com/watch?v=pucUyA42ZbQ&t=4s) 炫酷的演示（splashy demo）毫无用处，真正重要的是产生实际投资回报率（ROI）并为客户带来实质效益。 — Collin Campbell > [02:31](https://www.youtube.com/watch?v=pucUyA42ZbQ&t=151s) AI 的成熟度曲线衡量的不应是采纳率（Adoption），而是系统在故障后的恢复时间（Recovery Time）。 — Collin Campbell > [05:09](https://www.youtube.com/watch?v=pucUyA42ZbQ&t=309s) 模型并非在产生幻觉（hallucinating），而是在进行统计学上的改进与即兴发挥（statistically improving and improvising）。 — Collin Campbell > [10:46](https://www.youtube.com/watch?v=pucUyA42ZbQ&t=646s) 知识保留不再是一项合规任务，而是一项业务交付物（Knowledge retention isn't a compliance task anymore）。 — Collin Campbell > [12:09](https://www.youtube.com/watch?v=pucUyA42ZbQ&t=729s) 如果它无法向你证明自身有效或展示工作逻辑，就绝不上线（If it can't prove to you that it works or shows it works, don't ship it）。 — Collin Campbell > [18:23](https://www.youtube.com/watch?v=pucUyA42ZbQ&t=1103s) 集中护栏，去中心化想象力（Centralize the guardrails, decentralize the imagination）。 — Collin Campbell > [24:06](https://www.youtube.com/watch?v=pucUyA42ZbQ&t=1446s) 如果我们搞砸了，谁会第一时间发现？ — Collin Campbell > [26:51](https://www.youtube.com/watch?v=pucUyA42ZbQ&t=1611s) 我们不试图阻止错误的发生，我们追求的是从错误中快速学习，确保未来绝不再犯。 — Collin Campbell > [26:51](https://www.youtube.com/watch?v=pucUyA42ZbQ&t=1611s) 我不再衡量使用率了，去衡量你们学得够不够快、创造了什么实质价值。 — Collin Campbell > [26:51](https://www.youtube.com/watch?v=pucUyA42ZbQ&t=1611s) 洞察的速度远胜于上线的速度（Speed to insight beats speed to launch）……如果 AI 无法解释自己的推演逻辑，如果智能体说不清自己在干什么，你不过是在侥幸碰运气，它根本还不成熟。 — Collin Campbell > [31:11](https://www.youtube.com/watch?v=pucUyA42ZbQ&t=1871s) 有时候你需要做组（Reps）。 — Collin Campbell > [38:23](https://www.youtube.com/watch?v=pucUyA42ZbQ&t=2303s) 那些非常优秀、陪伴我走到今天的人，并不一定是带领业务走向未来的人选。 — Collin Campbell > [38:23](https://www.youtube.com/watch?v=pucUyA42ZbQ&t=2303s) 别管那个演示有多酷，它对长远发展毫无帮助。 — Collin Campbell ## Mentioned Collin Campbell, Cushman & Wakefield（戴德梁行）, Ai4, McKinsey（麦肯锡）, MIT（麻省理工学院）, Claude, Claude Code, OpenAI ChatGPT, Google Gemini, Cursor, Salesforce, Oracle, SharePoint, Pepsi（百事可乐）, Coca-Cola（可口可乐）, Iron Gwazi ## Video description Collin Campbell - Business Information Officer / Chief Information Officer - Recurring Revenue, Cushman & Wakefield AI projects rarely fail because of the models themselves. Instead, they fail because of messy data, flawed assumptions, and the unpredictable ways people use the technology. In this session, you'll hear how Cushman & Wakefield's AI strategy evolved through very real, and sometimes humorous missteps. We will dive into what went wrong, why it happened, and how each failure directly improved Cushman's approach to governance, adoption, and delivery speed. Attendees will walk away with practical lessons for deploying AI in complex enterprise environments where accuracy, trust, and scalability matter most. Expect clear takeaways you can use to move faster in the future by treating failures as fuel, not setbacks. [Source](https://www.youtube.com/watch?v=pucUyA42ZbQ)

[0:04] Collin Campbell: 炫酷的演示（splashy demo）毫无用处，真正重要的是产生实际投资回报率（ROI）并为客户带来实质效益。 [2:31] Collin Campbell: AI 的成熟度曲线衡量的不应是采纳率（Adoption），而是系统在故障后的恢复时间（Recovery Time）。 [5:09] Collin Campbell: 模型并非在产生幻觉（hallucinating），而是在进行统计学上的改进与即兴发挥（statistically improving and improvising）。 [10:46] Collin Campbell: 知识保留不再是一项合规任务，而是一项业务交付物（Knowledge retention isn't a compliance task anymore）。 [12:09] Collin Campbell: 如果它无法向你证明自身有效或展示工作逻辑，就绝不上线（If it can't prove to you that it works or shows it works, don't ship it）。 [18:23] Collin Campbell: 集中护栏，去中心化想象力（Centralize the guardrails, decentralize the imagination）。 [24:06] Collin Campbell: 如果我们搞砸了，谁会第一时间发现？ [26:51] Collin Campbell: 我们不试图阻止错误的发生，我们追求的是从错误中快速学习，确保未来绝不再犯。 [26:51] Collin Campbell: 我不再衡量使用率了，去衡量你们学得够不够快、创造了什么实质价值。 [26:51] Collin Campbell: 洞察的速度远胜于上线的速度（Speed to insight beats speed to launch）……如果 AI 无法解释自己的推演逻辑，如果智能体说不清自己在干什么，你不过是在侥幸碰运气，它根本还不成熟。 [31:11] Collin Campbell: 有时候你需要做组（Reps）。 [38:23] Collin Campbell: 那些非常优秀、陪伴我走到今天的人，并不一定是带领业务走向未来的人选。 [38:23] Collin Campbell: 别管那个演示有多酷，它对长远发展毫无帮助。

[10:46] Collin Campbell: If I pull up client Pepsi, I don't want to see all the information for Coca-Cola. [12:09] Collin Campbell: If it can't prove to you that it works or show it works, don't ship it. [15:29] Collin Campbell: I had a pilot that was 6 months long. It's not a pilot anymore. It's just a way of working at that point. [15:29] Collin Campbell: Centralize the guardrails, decentralize the imagination. [22:03] Collin Campbell: we'll just keep an eye on it [22:03] Collin Campbell: we'll just monitor it; it happened one time, it won't happen twice. [22:03] Collin Campbell: This is my idea. I love this thing. We're going to do it. [24:06] Collin Campbell: If we were to fail, who would notice? [29:40] Collin Campbell: We get so many free lessons by the things we screw up. Take notes. [29:40] Collin Campbell: Speed to insight beats the speed to launch. [29:40] Collin Campbell: If the AI can't explain itself... you're just getting lucky. It's not mature yet. [38:17] Collin Campbell: I don't care how cool that is. That ain't going to help.

[0:04] So, this talk's a little bit different.  Um, we've had a lot of very practical and [0:08] very technical conversations around kind of  what happened, what worked, what didn't work. [0:13] Um, I think we're all experts now because I'm day  three, right? So, what I would like to do here [0:18] is something a little different. I will leave  time for questions at the end. I feel like we [0:21] just don't get enough interaction. So, I'll do  that. But what we did is a little what I want [0:25] to do here is a little different. I want to talk a  little bit more about what worked for us. I had to [0:29] kind of sanitize a few things. You know, marketing  and PR had to go take a few things out. But what [0:34] we did is just have a real honest conversation  about what's going on. What did we learn from [0:38] and what did we do different? So, if you walk  away with something, I feel like I achieved. [0:43] So, who am I? Why should you listen to me? Uh,  so I'm Colin Campbell. I am the CIO of Recurring [0:47] Revenue over Kushman and Wakefield. Kushman and  Wakefield is a 65,000 person global real estate [0:52] company and I oversee the recurring revenue.  So that think think about running a building, [0:57] owning a building, managing that building, the  portfolio analysis, how much that thing is worth. [1:02] These are three to sevenyear agreements that  we signed where I have to win the business back [1:06] every day showing incremental value. So  splashy demo, not good practical value, [1:11] generating ROI and client benefit. Great. You'll  see a theme here coming by the way. So the idea [1:17] here is that we have to win this every day. We  have to have practical implications to AI. Well, [1:22] AI by its definition is impractical. You have  to find a way to give it the practicality to it. [1:28] So, a few other things about me. One, I am one  pragmatic and valuedriven. That basically means [1:34] that it's not cool that it looks cool. It is great  when it actually saves money, delivers value, [1:39] differentiation, and so forth. Second thing,  outside of work, I love theme parks. I love [1:45] Halloween. It's getting spooky season. Lowe's  just released all of their Halloween decorations. [1:50] Uh and then of course I also love a good roller  coaster. You'd heard that uh fact already. [1:56] So again, what's the journey for the day? What's  the agenda? Very, very simple. If I'm going to [2:00] hold you hostage for 30 minutes, I want to make  sure that you get something out of this. So what [2:04] are the patterns that we went through? What are  some key examples that went along with that? So [2:08] I'm going to share that through three stories.  The framework we established kind of go around [2:12] that. And then last but not least, the frame  the playbook that went along with that as well. [2:18] So kind of how did I take my intake process  governance and other practical applications [2:23] to say okay how do I not do it again and  how do I judge the next one based upon that. [2:31] All right I love a good stat. These are a  little dated and that makes them more fun for me [2:35] personally. We've all seen stats. It's usually the  third slide of everyone's presentation. I kind of [2:39] counted on that. So right now 88% of organizations  regularly use AI. That's from McKenzie. [2:45] There's no way that's not like 98% now just  by the time this has came out. Secondly, [2:50] 34% are reimagining workaround AI. So what are  the other 66% doing? Well, this is where I think [2:57] we'll see the crux of my conversation go. Is  there a difference between reimagining and [3:01] just getting incremental productive improvements?  Which one do you want to do and which one is the [3:05] right way for you? And then last, we have that  old famous MIT stat that we all saw last year [3:11] and I just like putting it up there. that says  95% of enterprise generative AI programs fail. [3:16] Interesting. I'd actually really like to know what  that statistic is now when you throw in aentic. [3:21] So, it's an interesting stat to go along with  it. Couple other things to kind of go with it. [3:27] This is my view. Maturity AI maturity  curve isn't adoption, it's recovery time. [3:33] How do I why do I feel that way? Very very  simple philosophy and I'll break it down to [3:37] three elements below. But if I gave 98%  of the labor of Kushman and Wakefield [3:43] a clawed or open AI chat GPT Google Gemini  license, would they instantly be better? [3:53] Yes. But did I get business value for it? Maybe.  That's the theory of this. It's not an adoption [4:00] curve. It's about learning what works and  then recovering off of it. So three basic [4:04] principles. Uh I did this presentation a little  backwards. I'll give you the answer first. So one, [4:10] every program is going to have an oops. That's  basically why we do this. It's experimentation. [4:15] How do I take that experimentation and then take  that to the next level? Secondly, we don't like [4:20] to learn from our mistakes, especially with AI  because it's so easy to jump to the next use case [4:25] like all right, oops, that didn't work. Hop  over to the next one, instantaneous value ad. [4:30] And then secondly, we don't view this in through  like traditional engineering methods because [4:34] the speed of which you can create is so much  faster. If we take a very basic instrumentation [4:39] approach into this, you get to a different  outcome which says okay why did I do this [4:45] and how do I how do I now turn that into a system  that prevents the next 10 times of this happening. [4:51] My stories are all driven around this problem  and said we did something it created a pretty [4:58] hilarious failure in some cases and we'll talk  about that and then afterwards how do you then [5:02] go in and do something different so you don't  have that problem again. practical application. [5:09] So, three patterns, three three different root  causes, all one opportunity to get better. [5:14] The first one, uh, this is one I think most  of us in the early adoption phases of AI [5:18] went through. If you're not, you could still be  living this today. I call it the ghost effect. [5:22] Uh, that lingering old piece of data, that PDF  no one ever took down, that org chart that's been [5:29] around forever, whatever it could have been, uh,  that was out there. And what that root cause was [5:35] we didn't actually think about we thought about  data cleanliness and data architecture and all [5:40] these different data governance principles  through the lens of traditional standardized [5:44] tabular databases. Cool. SQL was great. Salesforce  was great. Did you look at SharePoint? [snorts] [5:54] Practical, right? Second one, very simple one. The  confident fabrication. This is going to be a story [6:00] about one of our earlier agentic opportunities  where we didn't really understand how to create [6:06] the level of traceability within an agent and  within some early sandboxes had some pretty fun, [6:11] let's call it oopsies along the way. It wasn't  hallucinating. It was statistically improving and [6:18] improvising. I that was literally the answer  that was given back to us. I thought it was [6:26] I really hope my 14-year-old doesn't  learn that phrase, Dad. Like, dad, [6:29] I wasn't I didn't lie to you. I was  statistically improving and improvising. [6:34] And then last but not least was one that I think  should and I actually go I'll take a moment here. [6:38] Show of hands. Did anyone attend the kind of  unframed Kushman Wakefield talk at the beginning? [6:44] All five of you. Thanks. Uh but this is one that  came up on that is really controlling the pilot [6:49] the pilot swarm. One of the easiest things to  do is get overwhelmed. Whether it's through [6:53] good ideas in the field that are federated and  said, "Hey, look what I created." It could be [6:57] really smart vendors offering you different  services and capabilities. You kind of just [7:02] have like a potluck meal of AI pilots going on  everywhere, but nothing's really driving value. [7:08] They're feeding on your hunger for AI, not  the ability to drive your business forward. [7:16] So, let's go to the field stories.  some of the most fun for me. [7:22] So the chatbot that time forgot. So early on  everyone's probably one of your first two or [7:27] three use cases had to be about a chatbot. I want  to go put a very simple way to interact with our [7:35] standardized data sets so I can understand what  is the time off calendar or who do I call if I [7:41] need to do this thing. That answer wash rinse  repeat thing was like the easiest one all of [7:46] us saw. Instantaneous value ad. I can get that  into the field. We got an answer coming. Well, [7:52] we had a fun one. So, we rolled this out. It was  a very great I'm intentionally not mentioning [7:57] vendors or names here, individuals, but happy to  talk about our our existing tax stack afterwards. [8:02] Uh, but what we did is we rolled out a chatbot  that was there to do that, answered basic [8:07] questions. We were very proud of it. It was in  a very vis like a phase 2 phase three rollout. [8:13] And then on a presentation to the broader  technology team, someone asked it a very [8:18] basic organizational question around who  is the executive in charge of technology. [8:24] It gave my boss's predecessor as the answer.  Outside of the hilarity of the awkwardness, [8:31] it was actually really funny because then  we said, "Well, why do you think that?" [8:37] And it pulls up a little little PowerPoint  presentation. you click on that and sure enough [8:42] that was who the CEO or the uh the CDIO was in  2016 2017 not relevant to our current situation. [8:52] So it was really interesting. So the  first thing to take away from this is one [8:57] really think about the knowledge you're trying to  pull into your AI. We always talk about there's a [9:02] thousand tools to go do it, but fundamentally you  do have to think about what do I want to actually [9:08] ingest in this thing and does a PowerPoint  from 10 years ago really matter? Probably not. [9:15] So it was a really interesting thing around  just how you thought about data sets outside [9:19] of your core normalized data set of like you know  this comes from Salesforce or this is sitting in [9:23] a SQL database or an Oracle database. The funny  thing I actually came up with this is this idea of [9:31] of how do you think about data cleanup and  what can I clean and what should I clean [9:37] and this kind of started an interesting  journey for us is who owns that data. [9:43] So do you think about global process owners?  Do you think about domain owners? How do you [9:47] think about what is the person that can  say you know what the answer is Sal Kana [9:53] as opposed to the answer was whoever was in  2016. So, it was a very funny moment for me. [9:59] And then the other thing that really came out  of this that I thought was super interesting was [10:05] whose job is it to fix it because it's  all well and good to go and say, "Hey, [10:10] you know what? I found a problem." Or, "Hey,  you know what? That's really, really messy. [10:15] I have three kids. I certainly tell them to clean  their room and everything else, but it's really [10:20] funny to me because this was almost the corporate  equivalent of go clean your room." and my kid [10:25] looks at me and says that's not my job. So it  was really really funny for me. So what happened? [10:30] So all right so what did we do kind of bringing  this back down to it is we went in and said okay [10:35] what is the recency of data you want to include  within any search within this information [10:41] pretty big rule. Okay. All right let's just  say generically it was two years. Secondly [10:46] going along with that for where you have core  data domains who are the rightful owners. This was [10:52] an organizational chart. So, it's obviously the  people team, human resources, however your team, [10:56] however your company recognizes that group. Then  third, and kind of funny and most importantly, [11:03] how do I make sure that doesn't happen again  in a more critical environment to where I'm [11:06] actually talking to clients? When I pull up  client, you know, let's just use simple ones. [11:12] If I pull up client Pepsi, I don't want  to see all the information for Coca-Cola. [11:17] These are real problems you have to  start thinking about. So we really [11:20] started to really kind of organize our  data around what domain, how old is it, [11:24] and what's the recency around it and then  who can control what is the right answer. [11:33] So very very fun one on this. We took  principle one, very first principle we [11:40] ever came up with with an AI is knowledge  retention isn't a compliance task anymore. [11:46] And because we use such historical terms  like data governance, data stewardship, [11:52] data cleanliness. I'm sure we all have  various roles and teams around who do this. [11:57] This is now about knowledge, not just data.  So what is the company knowledge that we are [12:02] trying to retain and making sure that it's not  a compliance task, it is a business deliverable. [12:09] All right, the forecast that broke math. Um, this  one's a little personal for me because this was [12:16] me. So, uh, back when I was still coding and  having some fun with the early days of AI, I [12:24] we were creating early agents to go in and do very  basic revenue forecasts. So, based upon historical [12:30] trends, market factors, pipeline data, what's  X going to be in 3 months, 6 months, 12 months [12:40] should statistically be the most simplistic thing  you do. We probably have Excel sheets we all still [12:45] historically do this with, right? So before that,  [laughter] as I was trying to get out of that, [12:50] I was going in and saying, "Huh, this should  just be an automated thing. We type it in. We [12:55] know all this stuff. Boom. Go agent go."  So what happened was very, very simple. [13:01] It gave an answer that was plausible. It sounded  just a little okay, but just right enough. [13:11] Okay, that's good. We had no requirement at the  time for an agent to explain itself. Not truly [13:19] traceability but be going beyond that to basically  say essentially you know show your long division, [13:26] show your math, prove that you actually can  do this. And what happened when we did that [13:31] is we got a very very simple hilarious  answer. It was statistically simulated. [13:40] So, it had no idea how it came up  with the answer. It just seemed okay. [13:46] That's not really how we want to operate  our agents or our AI. That's not exactly [13:51] how we want to go about pretty much anything  where it just kind of hallucinated and it [13:55] it lied. And I found that to be a very intriguing  opportunity for us because it shows you that one, [14:03] think about your AI governance. Who  in your team can actually prove that? [14:08] Because again, to the naked eye, it looked  okay. But how it got there and how it used [14:15] the math and all the really clean data that  had all these beautiful owners assigned to it, [14:20] nope, it wasn't it didn't use any of that.  It kind of just made it up. Secondly, [14:25] when you think about the fix for this, how do  you make sure that this doesn't happen? So again, [14:31] we actually made it carry it. The output values  had to have a quantifiable number against it. [14:37] And then secondly, until we started certifying  data, which will be the next point, we had human [14:42] validations to make sure that this happened. We  weren't just letting numbers go out the door. [14:47] Good news, we caught this all in a sandbox  environment. We were all sitting there looking [14:50] at this kind of going, this is a very, very  interesting concept of how AI can be used. [14:56] But realistically, we had to  go back to the drawing board. [15:02] So very very principle tool of our development. So [15:07] was if it can't say prove to you that it works  or shows it works, don't ship it. It doesn't [15:14] it don't matter how cool the application  looks, how much productivity you're saving, [15:19] but if it didn't actually prove to you that 1  plus 1 equals 2, don't do it. Very, very simple. [15:29] This one's actually one of my favorites. So, the  swarm of AI pilots, and I really do think these [15:37] are like paper airplanes because like I felt like  the business resources that were assigned working [15:40] with me at the time were folding them up into  little airplanes and throwing in these ideas at me [15:45] all the time. And it was fascinating. You saw some  of the coolest projects you'd ever see in your [15:50] life around how you would digitize whole concepts  in commercial real estate, how you could totally [15:55] wake up tomorrow and not have to think about  anything. And then you started to realize that [16:00] every one of those had various problems. Some  of it's around adoption and change management. [16:04] Some of it's around enterprise scalability. Some  of it is that you don't actually think it does [16:08] what it says it does. Those level of compliance,  governance, and guardrails that we wanted to [16:13] put around it were very problematic because we  didn't have them yet. You had no consolidation, [16:18] no way to track measurable impact and  no portfolio ownership of the AI itself. [16:23] So what really happened to be as blunt as it can  is you had a lot of noise and a lot of expense [16:31] and hopefully someone out here can relate to this  scenario because it was one of the one of the more [16:36] ugly ones for us I'll say. So what have we been  doing? I will say I am very honest and open. [16:41] We're not perfect in any way, shape or form but  we we're getting there. And the first thing is [16:47] just putting a request registry together. Getting  people to say what they want to do was one of the [16:51] easiest things we ever did. Then stagegate your  pilots. When we when we first went into this, [16:58] I had pilots that were three and four months  old. I had a pilot that was 6 months long. [17:02] It's not a pilot anymore. It's just a way of  working at that point. Uh so we wanted to make [17:07] sure we actually put stage gates around our pilots  no more than 90 days. Lastly, forced decision-m. [17:15] It's really interesting, especially nowadays when  you can build and you can deploy so quickly. The [17:20] things that hold us up are business decision-m.  Can someone say yes? Can someone say no? Are they [17:26] empowered to say yes? Are they saying yes just for  their market, region, division, or are they saying [17:31] it across the board? Yes. Really getting that kind  of DOA, if you will, that delegation of authority, [17:37] right? That was actually the one of the hardest  parts of this is, okay, you told me what we got. [17:42] We've run it through this pilot. feel  good about it. Who gets to say yes? [17:47] And then lastly, we actually by doing this, we  put in a much stronger like what does it take to [17:52] go production level scale and it actually worked  out a lot better for us for two main reasons. One, [17:59] you had much stronger business buying and adoption  going forward. Number two, which I think is just [18:04] as equally as important, is you are able to  prevent duplicative time across the board. [18:10] So as we were sitting there going like why do I  have two things all looking at accounts payable? [18:13] Why do I have three things all looking at when  do I go dispatch an engineer or something? [18:18] It was very very simple to say like how can I  control and only get a be and get the best out [18:23] of what we're trying to do. So we came up with  this and this is not my line. This was actually [18:29] from my team so they get the full credit  for this is you centralize the guardrails. [18:34] here is the rules of the road. And  then you decentralize the imagination. [18:40] I don't actually think I know all the best things  for what we should do for tech in in real estate. [18:44] Oh well, maybe my wife thinks I do and maybe I do  sometimes secretly think I know the best. But in [18:49] reality, the answer here is to think about how you  can take the best of what's going on in the field [18:53] and the suggestions that come in and then apply  it your guardrails to it to ensure safe, secure, [18:59] fast, and valuedriven uh deployments. The other  thing that's not on here and why we did this is [19:06] for those of you who did not attend the my session  with unframe very very simple we were a buy shop [19:13] we were almost exclusively SAS we did not build  software that was not in my model of operating [19:20] we moved to a hybrid model where I am  sometimes building and I am sometimes buying [19:26] and when I want to do which one  is also part of the guardrails [19:30] that says I will maybe by a particular system  or a particular service to get that in there [19:37] so that I can move faster while still maintaining  a buy option to be able to decentralize [19:42] deployments and give attention to more a more  larger base of business users within my ecosystem. [19:48] So again, all these guard rails were about  building the right decision frameworks. [19:55] So, I do want to kind of get one into one  very simple and I didn't mean this to look as [20:00] buzzwordy as it did. That is not my intention. So,  apologies to the room for that. But one thing I [20:06] did want to say is build your learning loop. This  is mine. I don't think it's perfect in any way, [20:12] shape, or form. I will tell you, my team will  certainly can throw sticks at it all day long. [20:17] But what we want to do is basically  find a way to get to our failure faster. [20:24] Because every one of those failures I just talked  about is now a guardrail I don't worry about. [20:29] I don't worry that it's looking at six-year-old  data now. I don't worry about who is the owner [20:33] of that data now. Now I have advanced questions  that say things like can I certify that data? [20:38] So every time someone asks a question, it's  always that number or that piece of information. [20:43] So it's a much different model.  So it's incredibly simplistic. [20:48] Make sure you have the right tooling  and processes to catch your mistakes. [20:54] I did not at first I did not know that I needed  to be thinking about how to catch the mistake of [20:59] an AI. Secondly, trace it. Why did it do it? And  don't do like the little like, oh, okay, it kind [21:05] of did it this way. Like honestly figure out  like it's because we didn't have a document [21:09] retention policy is different than it looked  at an old document. That level of difference [21:13] allows you to then create the scalability  and enterprise awareness to go forward. [21:18] Lastly, you got to fix it. And fix it could be I  close it out. Fix it as you adjust the code. fix [21:22] it is you just delete the old document. But fixing  it is relevant. And then lastly, compound it. [21:28] And this is the secret sauce in my opinion. You  built the guardrail. Put that into your system. [21:35] Whether you're using clawed code and cursor,  whether you are just doing basic prompts, whether [21:40] you're creating a skill in chat GPT, make sure it  references your architecture and references these [21:46] decisions you've made across the board. Some of  us do this using various tools and systems. I'm [21:51] sure there's several vendors that can gladly sell  you one, but making sure that these guardrails are [21:57] always applied allows you to move even faster. I  will give you one stat that is not on this slide. [22:03] We went from having an average deployment time  of roughly for from AI from ideation to delivery [22:11] of about it used to be close to around two  to three weeks because we were constantly [22:15] having to go and revitalize and check that's  now about two days from idea to production. [22:25] So this is me getting a little hypothetical  here. So what did I what did I look at us and say [22:33] because we're not perfect. But if we were  perfect, what would I say we would do? [22:38] So sometimes we guess and guess there's  nothing wrong with guessing. And a company [22:42] that's guessing does very simple things. You're  not tracing. You're really kind of just getting [22:47] pilots to get market share or pilots to say you're  doing something. You're making the same mistakes [22:53] or you're fixing the same mistakes multiple times.  the old problem of old software coding or you ever [23:00] in a meeting and this is a quote from me you leave  a motoring with oh we'll just keep an eye on it [23:06] we'll just monitor it it's can't it happened one  time it ain't happened twice it ain't happened [23:10] four times to an enterprise that learns very  simplistically we trace it back to the data [23:17] it goes back to a document goes back to a  database goes back to an edge source system [23:22] every pilot has an owner who's the one person  that's going going to stand up and say, [23:25] "This is my idea. I love this thing. We're going  to do it." And then make sure you have every [23:30] misstep becomes the guardrail of the next time  for the next time, the next team, the next group. [23:35] We did that both within our sprint planning  sessions. We do that now within uh a little [23:40] bit more of a systemic approach through our  proprietary uh intake tools. And then lastly, [23:46] make sure the post-mortems actually happen and  then they actually occur with a system change [23:50] or a process change with it. That level of detail,  that level of involvement makes a very very simple [23:59] way forward. So this is my I leave for you.  If you were to go and we we were to open up [24:06] our proprietary app for intakes, these are  the eight questions that sit within there. [24:12] And notice that they're not very techy. They're  written so they can be consumed or or you know, [24:18] whoever's filling out the field can apply it.  So, not going to read all of them, but I just [24:22] want to hit a couple that matter the most. First  one to really focus in on within question one is, [24:28] is this really going to be grounded in data or is  this like a feel? That's a really strange question [24:33] because it's not always that answer. Sometimes  that you're trying to do a creative activity, [24:37] marketing, other things, but other times you want  it to be grounded in the market analysis for real [24:44] for you know rent in New York City for example.  Next one is visibility. If we are using data or [24:53] are we are doing something when's when's we  actually look at it? When do we care? Like [24:57] it's cool to say I want to do something but is  this net new? Are we doing parts of this today? [25:04] And then within the ownership, this is my  personal favorite one that came in recently is [25:10] if we were to fail, who would notice? And this  is done in two ways. First way is always done [25:18] internally, meaning it catches it, a little red  sensor goes off somewhere, you know, you think [25:23] back to the old movies of the blinking red light  of doom, something bad's about to happen. Uh, [25:28] but more realistically, as we start to deploy  things that could be more in the field facing [25:33] or even client face, how quickly can  you notice something's gone wrong? [25:39] And then lastly, the discipline  around it is how I always phrase it. [25:44] What would happen? How would I kill this thing  if I have to? How many bad things were going [25:49] with it? How are we orchestrating that? We  use these big words to basically say someone's [25:53] watching this thing, right? We have all these  fun words that we use now in AI, but someone's [25:57] watching the very simplistic agent going across  the board. What happens next? How do you know [26:01] if I have to stop it? How do I know the impacts  of that? How do I unwind it if I had to do it? [26:07] And then last but not least, if all that is  true and I can answer yes to all of that, [26:13] can I understand the logic and reasoning behind it  to then make any systemic changes to go after it? [26:21] Very simple. Eight questions saves that's  probably 18 months of us trying to learn [26:26] going forward. We put that on our intake  app. So, as someone is applying for an [26:31] for an application, our customuilt thing, they put  it in, they answer these eight questions with a [26:36] couple very simple like who are you questions  after that uh and we kind of move forward. [26:43] All right, I said I was going to leave  time for questions, so I'm not going to [26:46] I'm not going to uh stop on that. Few more things. [26:51] This is my general principle for my team and  I think it's good for a team of technology uh [26:57] engaged individuals whether you're a client, a  vendor or uh another technology leader out there. [27:05] We don't try to prevent mistakes. We try to learn  from them to just not have them in the future. [27:12] And how fast we do that increases our knowledge  base, which is great, but it also increases the [27:18] muscle memory of your team in your business. And  if you do that transparently, you also get trust. [27:24] So that same combination of speed, honesty is more  fun now and more relevant now than it's ever been. [27:32] And as we start to do this, we really try to  make sure that the understanding of the field, [27:38] the business, the consumer of whatever my team  is building understands that the partnership [27:43] that we have with them and also understands that  we are trying to learn as well as fast as we can. [27:50] All right. with 10 minutes left. Three  takeaways. One, never hide a failure. [27:59] If I'm crazy enough to put my name  on a slide that said AI failures, [28:05] I'm not going to put on the Christmas card,  guys. I'm not going to lie to you. But the uh [28:09] if I'm brave enough to do that, you're  brave enough to talk about your failures. [28:13] You should be brave enough to  understand how you learn from them. [28:16] You turn them into the tools that  allow you to move forward faster. [28:19] Lastly, this is my personal belief. I know there  will probably be people who disagree with this. [28:24] I don't measure utilization anymore. We used to  talk about how many licenses we shipped or how [28:31] what is the consumption of tokens or whatever.  Start asking if if are you learning fast enough? [28:38] Start measuring around value. Start and value  could be just the fact that we did something, [28:43] we learned from it, we're never going to do it  again. But really stop measuring usage. And then [28:49] scaling pilots. This is a pet peeve of mine. Uh,  and I've heard it done a little bit, but this is [28:54] just my personal view. We talk about how we scale  pilots. I had a pilot that went from five users [29:01] to 50 users or 50 users to 100 users. You want  to graduate a pilot. What does that mean? Is you [29:06] take the institutional scale and turn it into a a  viable product that is now living and breathing in [29:11] your ecosystem. And as more and more of us think  about our own AI, as we think about that agent [29:17] you build, as you're taking like, hey, this is how  I show up to work every day in a digital format. [29:23] The next version of this that's not on here  is think about that as your digital asset. [29:28] How are you going to treat that asset?  How are you going to nurture that asset [29:32] so that it continues to grow and be valuable  within within your IT ecosystem? So, three very [29:40] very very simple quotes of mine that I will again  just let everyone ingest. But most importantly, [29:50] we get so many free lessons  by the things we screw up. [29:55] Take notes. Then most importantly, my personal  belief is the speed to insight beats the speed [30:05] to launch. This is counterintuitive to how  all of us think about coding right now. [30:10] How we all think about the great idea can  get into the hands of the person beside us. [30:16] If it's not insightful, it's not valuable.  Doesn't matter that you are there first. [30:20] So, you need to do both, but you need to  be insightful all the time. And lastly, [30:27] if the AI can't explain itself, if the  agent can't explain what it's doing [30:32] and you don't understand what it is, you're  just getting lucky. It's not mature yet. So, [30:39] make sure you take that as a as a warning lesson  from us. As we learned across our ecosystem, [30:45] we built these guardrails in to protect us to  to basically guarantee the quality of service [30:51] that went along with it. So, with that said, I  appreciate [clears throat] you guys for listening [30:56] to me and hopefully that was not awful on 2 PM  on the last day of a conference after lunch. [31:04] So, if we have time for questions, I'm  more than happy to talk. [applause] [31:11] You sir, the front. I was curious  about the game that correct. [31:19] So sometimes you need reps is the way I look  at it and 90day is the maximum gate. So his [31:25] question is why would I say 90 days if I'm talking  about 2 to 4 day deployments. Great question. Uh [31:30] 90 days is the maximum day. We don't try to do  that but every now and then your AI needs reps. [31:35] Uh we talk about lease admin we talk about lease  abstraction within uh commercial real estate. [31:39] Lease abstraction just means contracts and reading  them digitally. Uh what we like to think about [31:44] with that is the first 10 okay the next 2,000  much better 10,000 in they're great I needed reps [31:52] I needed that bigger I it's almost like you're  going to the gym right you needed to get those [31:56] reps in so you had a repetition understanding  as well as you were actually training the model [32:00] along with it because sometimes if I would to  cut it off at day four I'd be like ah useless [32:07] uh you in the front row you I'm just  curious if there was like a culture shift [32:13] [laughter] [gasps] to get people on board with  these changes. Um, it's ongoing. Uh, I will say we [32:22] have had both a top down approach and a bottoms up  approach. Uh, the greatest way to drive a culture [32:27] shift for us was actually FOMO where people saw  what someone neat someone else did and they're [32:33] like, "Oh man, I I need to be able to do that  too. That's really cool." Um, the problem that [32:37] we had with culture shift is sometimes it went a  little too far. So, we went from like essentially [32:43] a command and control operation to a very much  let's all kind of figure this out together. [32:48] And then we almost went too far to where you  started to have potentially the way that person [32:52] A did their job and the way that person B did  their job not necessarily being a brand alignment. [32:58] So, we're still kind of figuring it out, but  most importantly for us was creating almost the [33:03] carrot versus the stick approach within driving  change. I saw Yes. I was wondering about register. [33:21] [laughter] [33:25] So I'm going to give you the very practical  answer that maybe not every company could do [33:29] but realistically the first thing we did is we  started auditing where expenses were going. So [33:33] where people were actually tracking expenses. Um  the second thing we did was we actually tracked uh [33:39] ingoing requests for data exchanges. So who was  actually trying to get a new pipeline or an API [33:43] access or something like that. So that's how  you kind of get got some of the unknown universe [33:48] from a pilot perspective and change. We  basically went in and said that you had a [33:52] kind of a singular funnel and that was supported  by leadership and management because they wanted [33:57] to scale these at an enterprise level. And then  what we've not done which is the upcoming change [34:03] is then really putting enterprise targets around  that. So then you would say how many times have [34:08] you done X Y or Z within that space. It's almost  like an automation target but that sounds way [34:13] more negative because it's actually about growth  not savings. Uh any other questions in the back? [34:19] Yes. This is a little bit related to culture one  you just mentioned. I hear you say, I say stick, [34:25] but I really like a couple of your your quotes up  there, particularly IT traffic control versus TSA, [34:33] things like that. Yeah. What lessons did you  have around putting those guards, you know, [34:36] put guard rail centralized? Correct. What lessons  did you have around putting those guards out? [34:43] Um, the first thing is realizing that not every  guardrail is going to be learned from a mistake. [34:47] Use that first guardrail to determine the next  three you should have already thought about. [34:51] uh from a cultural side of that  which I think is really interesting [34:56] is you started to really drive what was going  to be the value and how you determined value [35:01] because sometimes it is and I believe in  hard ROI meaning that if I do if I spend a [35:06] million did I make 10 right just simple math on a  Thursday afternoon when we got that as the belief [35:13] of the firm not the belief of the individual  that really was the big cultural shift from [35:20] how you started to control swarm how you started  to actually think about I have a role to play [35:25] but I'm not you know engineer app developer  QA you know judge and jury at the same time [35:33] so that was the bigger that was one of the bigger  elements within that ecosystem um the traffic cop [35:38] side of it is still interestingly problematic  because that does create it as a bottleneck [35:44] and no level of system in my view here  or system can control that because all [35:50] you can do is make the processes seamless but  you still have to account for enterprise risk [35:55] whether that's a brand risk whether that's  an infosc risk whether type of risk it is [36:00] change management risk so we still have that I  will be very honest and blunt with you but we're [36:05] working through it largely through uh largely  through just education and adoption and then [36:11] kind of having people understand what their role  using the enterprise tool sets are other questions [36:20] Yes. [36:22] [snorts] A lot [36:28] of what I've heard this week is  there's so much promise. We have [36:34] so much hope and it should allow us  to take great value and speed things [36:40] along and yet we paradox because a lot  of also we have to do a lot of things. [36:47] Yeah. [36:52] Yeah. So, it's a great paradox. I actually like  that word for to describe this. So, from a change [36:57] management perspective, when we first really  started to our eye journey and earn I mean, [37:03] we've had AI forever. Everyone's got like  their weird machine learning things, but like [37:06] we talk about real kind of, you know, light  bulb moments for AI. We did full education [37:12] across our executive suite to say this is the  opportunity. And I will tell you that went okay. [37:18] Uh the second thing we did that got  much more attention to them is we did [37:22] modeling of every terrible thing that  could happen in our industry with AI [37:26] from like full replacement or anything else for  that matter. That really got their attention and [37:32] then from there we started cascading it down into  different layers of basically executives on down [37:38] to really understand their involvement and role  within kind of the AI structure and strategy of [37:42] the firm. In some cases, the AI strategy  is please use our use our enterprise tools [37:49] because that's in a safe productive way for you  to go and use claude for example. In other cases, [37:54] it's you are working on enterprise data assets  across the firm that have registered owners. [38:00] We need to make sure we understand exactly what  that means for you to then be able to go and do a [38:05] level of action against them. So it was a little  bit of a journey but again it was very much an [38:10] education of the executives down into the  the uh different division lines and leads. [38:17] I have time for one more question if anyone is  interested. If not you will not hurt my feelings. [38:23] I got one. Go ahead. [snorts] That's the  question we haven't asked that we could have. [38:29] Oh that's that is a complete copout question.  What's the question someone should have asked? [38:35] Oh man, you could have asked like what was  my favorite roller coaster. That would have [38:38] been a cool one. Um but no, the uh and  the answer is Iron Guazian Tampa. Uh the [38:44] the right answer and right question is I'm all  if where am I in my journey? Like I will tell you [38:51] right now we are not perfect. We have not got we  don't have all of our data assets under control. [38:56] We don't have the most perfect AI strategy. I  think that's kind of funny when people kind of [39:00] talk like in this very transparent way like  oh we're perfect. Oh, great. That's amazing. [39:06] Wish I was. Uh, but to answer your question  specifically, what's the question I wish [39:10] I wish someone would ask? If you could go back  right now and say it's November of 2024 when all [39:17] of this really started impacting business, what  would be the two things you would do different? [39:24] The first one is I would activate my talent  strategy first. We were a little slow to [39:29] the game on that and that was a little bit dis  disappointing because and I could talk forever on [39:35] this one. I what it means to be an IT leader right  now is completely different than it was before [39:41] and therefore the people who are lovely and got  me here weren't the right people going forward. [39:46] So I would have activated my strategy  first, my people strategy first. Secondly, [39:51] I should have known some of those guard rails.  If you didn't look at my presentation and go, [39:55] "Yeah, duh. Come on, guy. Thought you were a CIO.  Um, that's kind of a funny one, though. But you [40:01] don't think about these things. You think about  speed. You think about deployment. You think about [40:04] this company over here is offering me something  amazing in AP. This company over here is [40:07] doing something completely differentiating in  digital buildings. You think about the shininess [40:11] instead of the actual programmatic approach to  creating a system that learns and goes forward. [40:17] So, I would go back and tell myself be  like, hey, I don't care how cool that is. [40:21] That ain't going to help. So, thank  you all for your time. [applause]