Faros AI Link Developer Session Data to SDLC Graph to Prove ROI
Ai4
AI usage expands pull request sizes by 51% and quintuples review times, forcing engineering teams to integrate session telemetry into software development graphs to eliminate AI waste.
## Background
This talk was delivered by Chase Norton, Head of AI at Faros, and published on YouTube by Ai4 on August 19, 2026. The presentation explores how engineering organizations can move beyond basic AI tool adoption and prove actual return on investment (ROI) by connecting AI session data directly to shipped software and business outcomes.
## TL;DR
While AI adoption has surged in software development, it has created new bottlenecks like longer pull request review times, making it necessary to evaluate AI spend based on real work outcomes rather than simple token usage. Faros introduces a framework that uses OpenTelemetry to pull raw AI session data into the standard SDLC graph, linking developer prompts directly to engineering artifacts and strategic business initiatives. This connected data allows engineering leaders to conduct counterfactual modeling, establish internal performance benchmarks, and run closed-loop governance through centralized gateway routing. By following the four pillars of Observe, Simulate, Review, and Govern, organizations can enforce policies that eliminate waste and maximize outcomes.
## Atmosphere & style
The presentation is analytical, pragmatic, and grounded in technical reality, delivered from the perspective of a hands-on practitioner. Chase Norton speaks with an authoritative yet accessible tone, blending real-world anecdotes—like commuting between Hawaii and San Francisco—with data-driven insights from thousands of developers. The experience feels like a structured masterclass in modern engineering management, cutting through AI hype to offer concrete architectural strategies for technical and executive leaders.
## Key takeaways
- [00:04](https://www.youtube.com/watch?v=c-YRayt-SGM&t=4s) The era of "token maxing" is over, and organizations are now under heavy pressure to prove return on investment (ROI) by connecting AI spend to real software outcomes.
- [00:04](https://www.youtube.com/watch?v=c-YRayt-SGM&t=4s) Faros's Whiplash Report revealed that while AI usage caused a 51% increase in pull request (PR) sizes, review times increased drastically (up to 5x), shifting the main engineering bottleneck downstream.
- [02:46](https://www.youtube.com/watch?v=c-YRayt-SGM&t=166s) Measuring ROI requires connecting three core components: Strategic Initiatives (Jira/Linear), Engineering Artifacts (GitHub/GitLab), and isolated AI Sessions.
- [02:46](https://www.youtube.com/watch?v=c-YRayt-SGM&t=166s) AI session data currently sits in a vacuum outside the traditional SDLC graph, making it necessary to pull telemetry directly into the standard SDLC workflow.
- [04:04](https://www.youtube.com/watch?v=c-YRayt-SGM&t=244s) The foundational step for tracking AI ROI is deploying an OpenTelemetry (OTel) collector to capture rich raw session traces and metadata beyond high-level provider dashboards.
- [05:59](https://www.youtube.com/watch?v=c-YRayt-SGM&t=359s) Mapping AI events to specific commits, PRs, and user stories creates an end-to-end trace from prompt activity to business epics, replacing guesswork with work evidence.
- [09:03](https://www.youtube.com/watch?v=c-YRayt-SGM&t=543s) Integrating AI telemetry into the SDLC graph enables counterfactual modeling to simulate whether tasks could be completed faster or cheaper using alternative models/harnesses.
- [09:03](https://www.youtube.com/watch?v=c-YRayt-SGM&t=543s) Generic public benchmarks are losing value compared to custom internal benchmarks, as top flagship models like Claude Opus or GPT-4.5 may not perform best on a company's specific codebase.
- [13:04](https://www.youtube.com/watch?v=c-YRayt-SGM&t=784s) By routing AI interactions through a centralized gateway, engineering organizations can enforce dynamic, policy-driven rules based on telemetry data.
- [13:04](https://www.youtube.com/watch?v=c-YRayt-SGM&t=784s) Faros's operational framework relies on four core pillars: Observe, Simulate, Review, and Govern to shift focus from token maxing to outcome maxing.
## Detailed walkthrough
### [00:04](https://www.youtube.com/watch?v=c-YRayt-SGM&t=4s) Shift from Token Maxing to Proving ROI
Chase Norton, Head of AI at Faros, opens his talk by sharing his perspective as someone who commutes back and forth between Hawaii and San Francisco. Reflecting on the past four or five years since "alien technology" arrived, he notes that while everyone has individually learned how to wield AI, the industry focus has fundamentally shifted. The era of "token maxing" seen in 2023 and 2024 has passed, and organizations are now under immense pressure to prove return on investment (ROI) and justify their spend with real outcomes. As a hands-on technical practitioner, Chase outlines Faros's practical implementation framework designed to connect AI activity to business outcomes from both the bottom up and top down.
Referencing Faros's *Whiplash Report*—a two-year study covering 22,000 developers across 4,000 teams—Chase highlights a major bottleneck shift in modern software development. While AI usage has led to a 51% increase in pull request (PR) sizes, the time required to review those PRs has surged drastically. Teams can now generate code rapidly, but the engineering bottleneck has simply moved downstream to the review phase, sparking debates around whether AI should also be used to automate code reviews. When multi-agent swarms can spin up a hundred agents at once and exhaust token budgets in minutes, companies must rigorously evaluate whether this spend produces anything of true organizational value.
- The *Whiplash Report* analyzed 22,000 engineers over two years to measure how AI adoption affects software engineering workflows.
- AI usage resulted in a 51% increase in pull request (PR) sizes.
- Reviewing PRs now takes approximately 5 times longer, shifting the development bottleneck to the review process.
[02:46](https://www.youtube.com/watch?v=c-YRayt-SGM&t=166s) The Three Components of the Software Engineering Graph
To measure ROI effectively, Chase breaks down the fundamental unit of optimization into tasks and outlines three core components that must be connected:
1. **Strategic Initiatives**: What leadership and the company care about, typically managed in project tracking tools like Jira or Linear through epics and tasks.
2. **Engineering Artifacts**: What developers actually produce, such as branches, commits, and pull requests in platforms like GitHub or GitLab.
3. **AI Sessions**: Interactive sessions, agent runs, and tool usages that represent modern AI activity.
Historically, the Software Development Life Cycle (SDLC) graph seamlessly linked strategic goals to engineering artifacts. However, AI sessions currently exist in an isolated vacuum outside the traditional SDLC. Chase emphasizes that the goal of Faros's framework is to pull AI session data directly into the standard SDLC graph, restoring the complete narrative from developer prompts to shipped business value.
[04:04](https://www.youtube.com/watch?v=c-YRayt-SGM&t=244s) Capturing Session Data with OpenTelemetry
The foundational step for any organization is setting up telemetry to collect raw AI session logs. Chase notes that the easiest, lowest-hanging fruit is deploying an OpenTelemetry (OTel) collector pointed at tools like Claude or Codex, which natively support batching and exporting rich metadata. Relying solely on basic cloud provider dashboards—which only show aggregate token spend and usage leaderboards—is insufficient because those high-level numbers miss crucial contextual details.
By capturing raw traces, metrics, and event logs, organizations can extract deep operational context embedded directly within the session payload. This low-level telemetry enables teams to trace precisely how developers interact with AI harnesses across different model providers and development environments.
- OTel collectors can be spun up quickly to capture native session logs from modern AI harnesses.
- Raw session logs capture deep metadata, including prompt payloads, response bodies, tool invocations, branches, PR IDs, and commit SHAs.
[05:59](https://www.youtube.com/watch?v=c-YRayt-SGM&t=359s) Connecting AI Telemetry to Strategic Initiatives
Once session telemetry is continuously captured, the next phase is establishing attribution by mapping AI events to specific engineering artifacts and tasks. Because OTel data is emitted as discrete events—accounting for the fact that AI interactions range from quick queries to multi-day sessions—these events contain key metadata signals like commit SHAs and branch names. These signals allow Faros to tie session activity directly to specific pull requests and individual developers.
After connecting AI sessions to GitHub or GitLab artifacts, traditional SDLC processing takes over to link those artifacts to Jira or Linear tasks. This creates an end-to-end trace: AI session data attaches to code changes, which attach to user stories, which aggregate into high-level business epics. Leaders can then evaluate whether an AI-assisted pull request actually closed a task, completed an epic, or contributed to a strategic priority that leadership cares about. Rather than guessing ROI, determining value becomes a standard data-modeling task based on explicit work evidence.
[09:03](https://www.youtube.com/watch?v=c-YRayt-SGM&t=543s) Counterfactual Modeling and Building Custom Benchmarks
Bringing AI session data into the core SDLC graph unlocks counterfactual "what-if" modeling. Because the exact model, harness configuration, prompt context, and agent configurations (such as `AGENTS.md` files) are recorded alongside the final work product, teams can simulate alternative scenarios. Engineering leaders can evaluate whether tasks could have been completed more efficiently using different models or lower-cost configurations—for instance, switching workload runs to specialized or discounted models without sacrificing code quality.
Furthermore, Chase argues that generic industry benchmarks are losing relevance compared to customized, internal benchmarks. Different engineering teams and codebases have unique requirements, meaning top-performing models on public leaderboards may not perform best on a company's specific repository. By analyzing real repository work, organizations can establish their own benchmark suites to determine which combination of models, harnesses, and context configurations yields optimal results for their actual code.
- Faros published its own open benchmarking results at `index.faros.ai`, testing models and harnesses against their internal codebases and tasks.
- Internal testing revealed that top-tier flagship models like Claude Opus or GPT-4.5 were not always the top performers for their specific workflows; alternative models (such as GLM-4.2) and open code harnesses frequently performed better and more cost-effectively.
[13:04](https://www.youtube.com/watch?v=c-YRayt-SGM&t=784s) Closed-Loop Governance with Policy-Driven Routing
Analyzing attributed SDLC data highlights clear patterns of efficiency and waste across an organization. When telemetry reveals that specific developer behaviors or tool configurations consistently yield higher-quality pull requests with less rework, organizations can codify these best practices into policy rules. By routing all developer AI interactions through a centralized AI gateway, companies can enforce dynamic, policy-driven rules across the entire engineering organization.
This structure creates a self-learning optimization loop: the gateway observes session activity, developers simulate and review outcomes, leadership adjusts routing policies based on real-world evidence, and the gateway automatically applies those optimizations across all teams. Chase summarizes Faros's framework around four key pillars: **Observe**, **Simulate**, **Review**, and **Governs**. Ultimately, moving beyond "token maxing" to "outcome maxing" requires capturing telemetry, integrating it into the SDLC, running counterfactual simulations, and establishing transparent ROI guardrails for both developers and executive leadership.
- **Observe, Simulate, Review, Govern**: The four pillars of Faros's AI management framework.
- Centralized gateway routing allows real-time policy enforcement across all engineering teams based on session telemetry and performance data.
## Video description
Chase Norton - Head of Artificial Intelligence, Faros
AI adoption is no longer the hard part; proving what it changed is. Engineering teams are now running AI-native software factories where tokens, agents, prompts, pull requests, incidents, reviews, and rework all move through different systems, while finance is asked to approve bigger AI budgets with little evidence of return. In this talk, I will show how to evaluate AI investment through work evidence: which AI activity reached shipped software, which work created leverage, and which spend became rework, overhead, or waste. We will use patterns from Faros' 2026 AI Engineering Report, based on two years of data from 22,000 developers and 4,000+ teams, to explain why adoption and throughput are not the same as ROI. Attendees will leave with a practical framework for tying AI dollars to shipped work, setting ROI guardrails, and continuously monitoring whether AI is creating capital or compounding waste. If you are accountable for AI investment and need a sharper answer than "we bought more tools," this session is built for you.
Faros AI将开发者会话数据关联至SDLC图谱以验证ROI
AI的使用使Pull Request规模增加了51%，审查时间增至五倍，迫使工程团队将开发会话遥测数据接入软件开发图谱以消除AI浪费。
尽管AI在软件开发中的应用大幅激增，但它也造成了Pull Request审查时间延长等新瓶颈，使得评估AI支出必须基于实际工作成果而非单纯的Token用量。Faros推出了一套框架，利用OpenTelemetry将原始AI会话数据接入标准SDLC图谱，直接将开发者的Prompt与工程产物及战略业务目标建立关联。这些打通的数据使工程管理者能够进行反事实建模、确立内部性能基准，并通过集中网关路由实施闭环治理。遵循“观察、模拟、审查与治理”这四大支柱，企业可以落地相应规范，从而消除浪费并最大化产出效益。
行业已告别“疯狂堆叠 Token”的阶段，企业面临着将 AI 支出与实际软件交付成果挂钩以证明投资回报率（ROI）的巨大压力。 (0:04)
Faros 的《Whiplash Report》指出，AI 的普及导致 PR 体积增加了 51%，但代码审查时间却大幅拉长了约 5 倍，将研发瓶颈直接转移到了下游。 (0:04)
追踪 AI 投资回报率的核心在于部署 OpenTelemetry（OTel）收集器，捕获原生会话日志，并将孤立的 AI 会话数据接入传统的 SDLC 图谱。 (4:04)
将 AI 事件映射到代码提交与业务需求后，团队可进行反事实模拟，通过自建内部基准测试来评估切换更高效或低成本模型的可行性。 (9:03)
通过集中式 AI 网关将最佳实践转化为动态路由策略，Faros 构建了涵盖“观察、模拟、评审、治理”的闭环优化管理框架。 (13:04)
Reading summary
Chase Norton, Head of AI at Faros, argues that software engineering has moved beyond experimental AI adoption to an era requiring clear proof of return on investment, which demands linking isolated AI session telemetry directly into the traditional software development lifecycle.
Downstream Bottlenecks and the ROI Challenge (0:04)
Norton notes that the initial phase of AI adoption, characterized by unchecked token consumption in 2023 and 2024, has given way to organizational demands for measurable business outcomes. As multi-agent systems and developer tools consume budgets rapidly, engineering leaders face pressure to verify whether higher output translates to meaningful organizational value.
To illustrate the operational impact of unmeasured adoption, Norton cites Faros's Whiplash Report, a two-year study of 22,000 developers across 4,000 teams. The research found that while AI assistance increased pull request (PR) sizes by 51%, the time required to review those PRs surged approximately fivefold. Rather than accelerating delivery overall, AI code generation largely shifted the development bottleneck downstream into the review phase.
Integrating AI Telemetry into the SDLC Graph (2:46)
Traditional software development lifecycle (SDLC) graphs link strategic initiatives in project management tools like Jira or Linear to engineering artifacts such as commits and pull requests in GitHub or GitLab. Norton observes that AI sessions—including interactive prompts and autonomous agent runs—currently operate in an isolated silo disconnected from this framework.
To bridge this gap, Faros advocates deploying OpenTelemetry (OTel) collectors targeting AI environments like Claude and Codex. Unlike high-level cloud provider dashboards that merely report aggregate token spend, OTel logs capture granular metadata such as prompt payloads, tool invocations, branch names, PR identifiers, and commit SHAs. Embedding this telemetry allows organizations to map discrete AI sessions directly to code artifacts and parent epics, establishing empirical attribution between model spend and completed business tasks.
Counterfactual Modeling and Repository-Specific Benchmarks (9:03)
Capturing full session parameters—including agent configurations, context files, and model harnesses—allows teams to conduct counterfactual simulations. Engineering leaders can model whether historical tasks could have been achieved with comparable quality using alternative configurations or lower-cost models.
Norton asserts that generic industry leaderboards are insufficient for predicting real-world repository performance. Based on tests published at index.faros.ai, internal evaluations showed that flagship models like Claude Opus and GPT-4.5 were not consistently the best fit for specific internal codebases. Instead, alternative models such as GLM-4.2 combined with open code harnesses frequently proved more effective and economical for their actual workflows.
Closed-Loop Governance Through Gateway Routing (13:04)
Faros structures its management framework around four operational pillars: Observe, Simulate, Review, and Govern. Once telemetry identifies which developer behaviors and tool configurations consistently produce clean code with minimal rework, companies can formalize these insights into operational rules.
By routing developer AI traffic through a centralized gateway, organizations can enforce dynamic, policy-driven model selection and guardrails in real time. Norton argues that this closed loop allows teams to systematically eliminate wasteful token usage, optimize routing based on empirical performance, and align day-to-day coding assistance with executive business objectives.
Norton concludes that establishing AI ROI requires moving from raw token usage to evidence-based outcome tracking, though this operational transition depends entirely on an organization's capacity to instrument granular telemetry and enforce centralized routing policies across fragmented developer environments.
精炼阅读摘要
Faros 人工智能负责人 Chase Norton 指出，软件行业已从盲目消耗算力的“Token 最大化”时期，转向必须以实际业务成效证明投资回报率（ROI）的新阶段。Faros 提出了一套实践框架，旨在通过深度遥测技术将孤立的 AI 会话数据纳入传统软件研发全流程图谱，从而建立从代码生成到业务价值的客观度量体系。
研发瓶颈后移与投资回报压力 (0:04)
在 2023 年至 2024 年期间，行业普遍关注模型调用与 Token 的消耗规模，但当前企业管理层面临着严苛的成本论证与产出证明压力。Norton 援引 Faros 历时两年、针对 4000 个工程团队共 22000 名开发者展开的《Whiplash Report》调查指出，AI 工具的普及虽然加速了代码编写，使拉取请求（PR）的代码体积平均增加了 51%，但同时导致团队审查这些 PR 的时间大幅延长至原先的约 5 倍。代码生成速度的提升并未消除工程交付阻碍，而是将核心瓶颈转移到了下游的代码审查阶段。
Norton 强调，当多智能体集群（Multi-agent Swarms）能够在数分钟内启动上百个 Agent 并耗尽 Token 预算时，企业如果无法评估这些代码变动是否具备实质组织价值，就无法验证智能化转型的实际收益。行业内目前围绕是否应进一步引入 AI 自动化审查展开了讨论，但根本矛盾依然在于如何将高昂的算力开销转化为可量化的业务交付成果。
将 AI 会话接入研发流程图谱 (2:46)
为了客观衡量 ROI，Faros 将优化的基本单元拆解为具体的工程任务，并梳理了软件工程图谱中必须相互关联的三大核心要素：由 Jira 或 Linear 追踪的战略举措（包括 Epic 与任务）、由 GitHub 或 GitLab 记录的工程产物（包括代码分支、提交记录与 PR），以及涵盖交互与工具调用的 AI 会话数据。Norton 指出，传统软件开发生命周期（SDLC）已经能够顺畅连接战略目标与代码产物，但现代 AI 会话却长期脱离该体系，处于孤立运行的数据真空状态。
解决该割裂的基础在于建立底层遥测体系。企业可以通过部署 OpenTelemetry（OTel）收集器，直接从 Claude 或 Codex 等支持批量导出丰富元数据的 AI 工具中捕获底层日志。单纯依赖云厂商提供的高层看板只能看到聚合的 Token 消耗和使用排行榜，缺乏关键上下文；而 OTel 能够捕获完整的 Prompt 载荷、模型响应体、工具调用细节、分支名称和 commit SHA。依托这些元数据，系统可以将分散的 AI 事件精准映射至具体的 PR 及背后的用户故事与业务 Epic，构建出从单次提示词到终端业务价值的完整因果链条。
反事实模拟与定制基准测试 (9:03)
当 AI 会话与工程产物实现完整关联后，企业便具备了进行反事实（what-if）推演的数据基础。由于系统完整记录了生成代码时的特定模型、外围封装配置（Harness）、提示词上下文以及 Agent 配置（如 AGENTS.md 文件），工程管理层可以针对历史任务发起模拟推演，评估若在当时采用替代模型或低成本配置，是否能够以更低的算力开销达成同等质量的代码产出，从而消除过度开销。
此外，Norton 提出行业通用基准测试正在失去对企业实际落地的指导意义，因为公开榜单无法反映特定私有代码库的具体规范与工程环境。Faros 在公开测试平台 index.faros.ai 上验证了这一观点：在其自身代码库和真实任务的测试中，诸如 Claude Opus 或 GPT-4.5 这类顶级旗舰模型并不总能取得最佳表现，反而是 GLM-4.2 等替代模型结合开源 Harness 在特定工作流中展现出了更高的性价比与有效性。企业需要基于自身代码资产构建定制化的基准测试套件，以甄选最契合自身业务的组合配置。
策略驱动网关与闭环治理机制 (13:04)
通过对归因后的 SDLC 数据进行深度分析，企业能够识别出组织内部的高效开发范式与潜在浪费路径。当遥测数据显示某种特定操作行为或工具配置能显著产出返工率更低、质量更高的 PR 时，管理层即可将这些最优实践沉淀为具体的工程规则，并通过集中式 AI 网关动态实施全员策略分发。
Faros 将该治理体系归纳为四个核心支柱：观察（Observe）、模拟（Simulate）、审查（Review）与治理（Govern）。网关负责实时监控开发者与模型的交互流量，工程团队对输出结果进行审查与模拟评估，管理层根据实证指标持续调整路由策略，再由网关自动将优化后的策略推行至所有工程团队。通过建立策略驱动的动态路由机制，企业得以构建自学习的优化闭环，最终实现从算力消耗向产出最大化的转变。
Norton 总结认为，破解 AI 投资回报率难题不能依赖脱离上下文的粗颗粒度消耗看板，而必须依靠基于 OTel 的精细化遥测、端到端研发图谱归因以及定制化基准测试来建立透明的治理规则。不过，如何确保策略网关在复杂的多团队工程场景下持续适应动态变化的代码审查标准，仍是组织规模化应用中需要长期应对的实践课题。
Detailed notes — zh
## Background
本内容由 YouTube 频道 Ai4 于 2026 年 8 月 19 日发布，主讲人为 Faros 的 AI 负责人 Chase Norton。视频围绕 engineering 团队如何从单纯的“刷 Token”工具尝鲜阶段，转向通过工作实证（work evidence）与全链路度量框架来证明 AI 的实际投资回报率（ROI）。
## TL;DR
Faros 的 AI 负责人 Chase Norton 指出，当前 AI 采纳已非难事，但证明 ROI 成为核心痛点——AI 加速代码生成的同时，导致 PR 审查时间暴增 5 倍并将瓶颈推向下游。他提出将任务作为最小优化单元，通过 OpenTelemetry 采集 AI 会话并深度关联 GitHub 代码工件与 Jira 业务目标，打破数据孤岛。团队可基于真实任务开展假设分析（What-ifs）与私有基准测试，最终借助 AI Gateway 实现策略驱动的动态路由与组织级自学习治理闭环。
## Atmosphere & style
演讲整体呈现出冷静、务实且极具洞察力的工程师专家视角。Chase Norton 以自身在夏威夷与硅谷之间通勤的从容姿态开场，跳脱了行业浮躁的炒作喧嚣，直击技术管理者面临的真实财务与交付痛点。整场分享逻辑严密、层层递进，用详实的数据与系统架构图化解团队直觉与现实之间的脱节感，给观众带来一种条理清晰、极具实操指导价值的笃定体验。
## Key takeaways
- [00:04](https://www.youtube.com/watch?v=c-YRayt-SGM&t=4s) AI 采用阶段已从最初的尝鲜与“刷 Token（token maxing）”转向证明投资回报率（ROI），技术管理层必须向董事会和业务方清晰量化 AI 投入对业务成果的实际贡献。
- [00:04](https://www.youtube.com/watch?v=c-YRayt-SGM&t=4s) 基于 2.2 万名开发者的《Whiplash 报告》显示，AI 虽加快了编码产出（PR 体积增加 51%），但导致代码审查时间暴增 5 倍，工程瓶颈向下游严重漂移。
- [00:04](https://www.youtube.com/watch?v=c-YRayt-SGM&t=4s) 并发运行的 Agent Swarm 会在短时间内消耗大量 Token，但 Token 消耗量并不等同于公司实际交付的商业价值。
- [02:46](https://www.youtube.com/watch?v=c-YRayt-SGM&t=166s) 准确衡量 AI 效能的关键在于将任务（Task）作为最小优化单元，打破 AI 会话作为数据孤岛的现状，构建统一的 SDLC 图谱。
- [02:46](https://www.youtube.com/watch?v=c-YRayt-SGM&t=166s) 打通图谱的核心在于将战略举措（Jira/Linear 上的 Epics）、工程工件（Git 分支、Commit SHA、PR）与 AI 会话三者进行端到端关联。
- [02:46](https://www.youtube.com/watch?v=c-YRayt-SGM&t=166s) 部署开源 OpenTelemetry（OTel）Collector 是最具实操性的遥测手段，可提取包含 Branch、Commit SHA、Prompt 及 Tool 调用的丰富会话元数据。
- [09:03](https://www.youtube.com/watch?v=c-YRayt-SGM&t=543s) 基于深度归因数据，团队可回溯历史执行“假设情景分析”（What-If Scenarios），模拟更换模型、微调 Prompt 或调整 Harness 架构对成本和交付质量的影响。
- [11:40](https://www.youtube.com/watch?v=c-YRayt-SGM&t=700s) 公开的通用大模型排行榜对真实工程指导价值有限，工程师直觉往往偏好顶级大模型，但基于自身代码库的专属基准测试常证明高性价比模型表现更优。
- [13:04](https://www.youtube.com/watch?v=c-YRayt-SGM&t=784s) 通过全链路数据可识别开发者的行为模式分化，找出高效交付路径与低效冗余模式，为企业级治理提供事实依据。
- [13:04](https://www.youtube.com/watch?v=c-YRayt-SGM&t=784s) 将所有 AI 流量收敛至统一的 AI Gateway，通过策略驱动路由（Policy-driven Routing）动态下发最佳配置，形成组织级的自学习优化闭环。
- [15:09](https://www.youtube.com/watch?v=c-YRayt-SGM&t=909s) 实现 AI 价值最大化（Outcome Maxing）包含三大核心步骤：捕获会话日志、深度关联 SDLC，以及基于效能数据进行动态路由与治理。
## Detailed walkthrough
### [00:04](https://www.youtube.com/watch?v=c-YRayt-SGM&t=4s) AI 采用之后的真正挑战：从“刷 Token”转向投资回报率（ROI）
Chase Norton（Faros 的 AI 负责人）开场分享了自己的背景：他生活在夏威夷，平时在夏威夷与旧金山之间往返通勤，这种环境让他能够跳出硅谷狂热的日常喧嚣，更冷静地审视 AI 的发展轨迹。他指出，人类接触大模型这种“外星科技”已有四五年时间，大家都在摸索如何驾驭并从中获取预期成果。然而，行业阶段已经发生显著变化：2023 年至 2024 年主要是“刷 Token（token maxing）”和工具尝鲜阶段，而当前工程团队和管理层必须直面 ROI（投资回报率）的量化与证明，向董事会和业务方明确说明 AI 投入到底带来了哪些实际业务成果。
为了解答 AI 对工程交付的真实影响，Faros 曾发布过一份基于 22,000 名工程师、跨度长达两年的《Whiplash 报告》（Whiplash Report）。研究发现，AI 工具的大规模使用确实大幅提升了编码产出速度，但也带来了一系列意想不到的副作用：
- **代码审查瓶颈转移**：报告显示 Pull Request（PR）的代码体积平均增加了约 51%，但最严重的问题在于 PR 的审查时间延长了约 5 倍（5x longer）。
- **工程瓶颈向下游漂移**：代码生成速度变快，导致下游工程师不堪重负；虽然部分团队试图探索“让 AI 审查 AI 生成的 PR”，但这在工程界引发了极大的争议与激烈讨论。
- **Token 消耗不等于商业价值**：随着现代 Agent 运行框架（Harness）支持一键并发启动上百个 Agent 的集群（Agent Swarm），团队很容易在极短时间内烧光 Token 预算，但这并不意味着这些消耗转化为了公司真正关心的交付成果。
```mermaid
flowchart LR
accTitle: AI 引入后软件工程交付瓶颈的转移
subgraph Production["代码生成阶段 (加速)"]
A["AI Agent Swarm / 编码助手"] --> B["代码产出激增<br/>(PR 体积 +51%)"]
end
subgraph Review["代码审查阶段 (新瓶颈)"]
B --> C["人工/自动化审查<br/>(耗时暴增 5 倍)"]
C --> D["合入与交付成果?"]
end
```
---
### [02:46](https://www.youtube.com/watch?v=c-YRayt-SGM&t=166s) 贯通 SDLC 图谱：建立战略任务、工程工件与 AI 会话的三维关联
为了客观衡量 AI 的真实产出，必须将任务（Task）确立为最小优化单元。传统的软件工程生命周期（SDLC）拥有成熟的建模体系，而当前最大的脱节在于——AI 会话（AI Sessions）被孤立在传统 SDLC 体系之外，成了一个数据孤岛。Chase Norton 提出，必须将以下三个核心要素深度连接并纳入统一的 SDLC 图谱中：
1. **战略举措（Strategic Initiatives）**：公司战略关注的重点目标，通常在 Jira 或 Linear 等项目管理工具中通过史诗（Epics）和任务（Tasks）进行层级化管理，代表了领导层的业务诉求。
2. **工程工件（Engineering Artifacts）**：工程师日常交付的代码实体，如 GitHub/GitLab 上的分支、Commit SHA 以及 Pull Requests。
3. **AI 会话（AI Sessions）**：工程师调用模型、Prompt、工具链进行交互的完整上下文记录与执行痕迹。
建立这种端到端关联的关键第一步是**数据遥测（Telemetry）**。当前最具实操性且立竿见影的低垂果实（Lowest-hanging fruit），就是在公司内部部署开源的 OpenTelemetry（OTel）Collector。诸如 Claude Code 等主流 Agent 框架均已原生支持通过 OTel 将运行会话以事件（Events）形式批量导出。
仅仅查看大模型厂商后台的 Token 消耗看板或排行榜是远远不够的，因为那只反映了表层开销。OTel 导出的会话日志中蕴含着丰富的元数据，包括：
- 每次会话关联的代码分支名、Commit SHA、PR 编号；
- 提示词（Prompts）、模型返回的 Response Body 以及具体调用的 Tools；
- 工程师 ID、Traces 与跨系统的执行指标。
```mermaid
flowchart TD
accTitle: AI 会话与传统 SDLC 的归因与关联架构
subgraph DataLayer["1. 遥测与采集"]
A1["AI Sessions / Harness<br/>(Claude Code, Codex 等)"] -->|OTel Events| A2["OpenTelemetry Collector"]
end
subgraph ArtifactLayer["2. 工程工件与代码库"]
A2 -->|提取 Branch / Commit SHA / PR| B1["代码托管平台<br/>(GitHub / GitLab)"]
end
subgraph StrategyLayer["3. 业务与战略目标"]
B1 -->|关联 Issue / Epic| C1["项目管理工具<br/>(Jira / Linear)"]
C1 --> C2["公司战略举措<br/>(ROI 与业务成果)"]
end
```
一旦在 OTel 事件日志中提取出 Commit SHA 和分支信息，AI 会话就能顺理成章地归因到 GitHub 的 PR 上；随后再通过 GitHub 与 Jira 的原生映射，将 AI 消耗无缝打通至业务 Epic 和战略目标中。至此，管理层便能清晰复盘：每次 AI 调用的开销究竟对应哪项战略任务？该 PR 最终是否成功合入？该功能是否真正落地交付？
---
### [09:03](https://www.youtube.com/watch?v=c-YRayt-SGM&t=543s) 基于真实归因构建“假设情景”（What-If Scenarios）与专属基准测试
当 AI 会话与工程工件、业务任务完成深度归因后，团队不仅能事后审计，更能获得一项强大的能力——**“回溯时间并构建假设情景（What-If Scenarios）”**。因为你精确掌握了完成特定工程任务所使用的模型名称、Harness 架构、Prompt 设定、上下文窗口乃至 `agents.md`（或 `agents.mmd`）配置文件，团队可以针对历史任务模拟各种优化策略：
- **模型与成本置换模拟**：如果团队在某类特定任务中不使用高昂的顶级模型（如 Claude Opus 或 Fable），而是切换到性价比更高的模型（例如降价 80% 的模型版本或更轻量的开源权重模型），能否在保证 PR 合入质量的前提下大幅削减成本？
- **架构与 Harness 调优**：调整 Agent 交互轮次、Context 策略或工具链配置后，研发效率与消耗比例会发生怎样的改变？
Chase Norton 强调，虽然公开领域的通用大模型基准测试（Benchmarks）层出不穷（从各类前沿大模型到 Kimi K3、GLM 系列开源权重模型等），但这些通用榜单对实际生产工程的指导意义正在边际递减。行业现在真正需要的是基于自身代码仓库（Repo）、业务特征与工程团队习惯所建立的**私有基准测试套件（Suite Benchmarks）**。通过在真实任务归因图谱上运行 What-If 假设分析，企业才能真正摸索出最适合自身研发体系的模型与 Harness 黄金配置组合。
### [11:40](https://www.youtube.com/watch?v=c-YRayt-SGM&t=700s) 构建企业自身的 AI 基准测试与假设分析（What-ifs）
单纯追踪 AI 的产出结果（Outcomes）只是第一步。当团队能够串联并完整讲述 AI 交互全过程的因果故事时，就可以进一步开展“假设分析”（What-ifs），并建立专属于企业自身代码库的 AI 基准测试。工程师通常有一种职业本能——倾向于直接选择市面上最新、最大、最快的模型，想要亲自体验新技术并在脑海中建立对新模型的认知。然而，通过将真实的工程任务、代码库和历史需求投入基准测试后，Faros 团队发现直觉往往并不可靠。
在 Faros 自家的实践中（公开展示于 index.faros.ai），他们将基准测试运行在内部的实际代码仓库与真实任务上。测试结果显示，在排行榜上名列前茅的往往并不是 Claude Opus 这类顶级巨无霸模型，一些轻量或特定适配的模型在特定任务中的表现和对齐度反而更高。这证明了脱离自身业务场景去盲目追捧大模型很容易造成资源浪费，企业必须依据自身的代码上下文建立真实数据驱动的评估体系。
- **基准测试透明化**：Faros 在 index.faros.ai 上公开了针对真实研发任务的基准评估数据与模型排行榜。
- **直觉与数据的冲突**：工程师倾向于选用最知名、参数量最大的前沿模型，但企业专属基准测试证明，针对特定代码库的最佳性价比模型往往与“直觉预设”大相径庭。
---
### [13:04](https://www.youtube.com/watch?v=c-YRayt-SGM&t=784s) 闭环治理：基于 AI Gateway 的策略驱动路由与组织级自学习优化
通过贯穿整个研发链条的会话追踪与基准测试，管理者能够清晰捕捉到工程师在使用 AI 时出现的行为模式分化。例如，团队中采用 X 模式的开发者所产生的会话更加高效、交付成果更多，而采用 Y 模式的开发者则存在大量冗余与低效消耗。基于这些洞察，企业便可以制定针对性的路由策略（Policies），并将其沉淀到统一的 AI 网关（AI Gateway）中。
```mermaid
flowchart TD
accTitle: AI 网关自学习优化闭环
accDescription: 展示从开发者 AI 流量经过网关、捕获并关联 SDLC、分析效率差异，再到策略反哺网关的自学习治理闭环。
subgraph SW_GW["AI 接入与执行层"]
A["开发者与 Agent 请求"] --> B["Faros AI Gateway<br/>(统一网关路由)"]
B --> C["多模型服务后端<br/>(LLMs / Code Tools)"]
end
subgraph SW_ANALYTICS["分析与洞察层"]
B -.->|"捕获日志与会话"| D["连接 SDLC 全链路"]
D --> E["批处理与假设分析<br/>(What-if 仿真)"]
E --> F["识别低效行为与高效模式"]
end
subgraph SW_POLICY["治理与策略闭环"]
F --> G["动态调整路由与护栏策略"]
G -->|"下发策略配置"| B
end
```
当所有 AI 流量统一流经 AI 网关并应用“策略驱动路由”（Policy-driven Routing）后，企业就拥有了绝对的掌控力。随着系统不断对事件进行批处理分析、运行假设仿真，系统能自动识别冗余行为与成功路径，进而动态调整网关配置。这种优化不再局限于个别工程师的微调，而是直接赋能并改变整个工程组织的 AI 使用方式，形成一个“观察、仿真、审查、治理”（Observe, Simulate, Review, Govern）的组织级自学习闭环。
- **行为模式比对**：通过全链路数据识别高效会话与低效行为的差异特征，为策略制定提供事实依据。
- **全流量统一接入**：将所有工程 AI 流量收敛至 Faros AI Gateway（或自建网关），实现集中管控。
- **动态策略下发**：网关配置随分析结果动态迭代，将个别最佳实践一键推广至全员，从源头阻断低效浪费。
---
### [15:09](https://www.youtube.com/watch?v=c-YRayt-SGM&t=909s) 走向“结果最大化”：连接 SDLC、沟通 ROI 与持续进化
在传统软件工程中，无论是成长为优秀工程师还是技术管理者，都有清晰成熟的发展路径；但在 AI 时代，所有人都在各自的电脑前摸索前行，缺乏现成的行业标准指南。因此，团队之间共享框架与方法论至关重要。实现 AI 价值最大化（Outcome Maxing）的核心在于三步走：首先捕获完整交互日志，其次将其深度连接到软件开发生命周期（SDLC）中以主动挖掘低效与浪费，最后通过动态网关进行策略路由与持续调优。
这套体系能够帮助技术团队以完全透明的方式向管理层和财务部门清晰汇报 AI 的真实投资回报率（ROI），同时也让一线工程师能清晰看到自身的效能表现并持续自我改进。演讲最后，Chase Norton 呼吁大家访问 faros.ai 和 index.faros.ai 交流探索，共同构建基于证据的 AI 研发度量体系。
- **捕获日志（Capture Logs）**：完整记录 Prompt、Agent 交互、Token 消耗及会话元数据。
- **关联 SDLC（Connect to SDLC）**：将 AI 行为映射至 PR、代码审查、CI/CD 构建、事故和返工，100% 暴露隐藏的研发瓶颈。
- **动态路由与治理（Dynamic Routing）**：基于效能数据动态切换模型与流程，实现工程产出最大化（Outcome Maxing）并向高层清晰证明 ROI。
## Notable quotes
> [00:04](https://www.youtube.com/watch?v=c-YRayt-SGM&t=4s) 人类接触大模型这种“外星科技”已有四五年时间，大家都在摸索如何驾驭并从中获取预期成果。然而，行业阶段已经发生显著变化：2023 年至 2024 年主要是“刷 Token（token maxing）”和工具尝鲜阶段，而当前工程团队和管理层必须直面 ROI（投资回报率）的量化与证明 — Chase Norton
> [00:04](https://www.youtube.com/watch?v=c-YRayt-SGM&t=4s) 报告显示 Pull Request（PR）的代码体积平均增加了约 51%，但最严重的问题在于 PR 的审查时间延长了约 5 倍（5x longer）。 — Chase Norton
> [00:04](https://www.youtube.com/watch?v=c-YRayt-SGM&t=4s) 随着现代 Agent 运行框架（Harness）支持一键并发启动上百个 Agent 的集群（Agent Swarm），团队很容易在极短时间内烧光 Token 预算，但这并不意味着这些消耗转化为了公司真正关心的交付成果。 — Chase Norton
> [02:46](https://www.youtube.com/watch?v=c-YRayt-SGM&t=166s) 传统的软件工程生命周期（SDLC）拥有成熟的建模体系，而当前最大的脱节在于——AI 会话（AI Sessions）被孤立在传统 SDLC 体系之外，成了一个数据孤岛。 — Chase Norton
> [02:46](https://www.youtube.com/watch?v=c-YRayt-SGM&t=166s) 当前最具实操性且立竿见影的低垂果实（Lowest-hanging fruit），就是在公司内部部署开源的 OpenTelemetry（OTel）Collector。 — Chase Norton
> [09:03](https://www.youtube.com/watch?v=c-YRayt-SGM&t=543s) 虽然公开领域的通用大模型基准测试（Benchmarks）层出不穷……但这些通用榜单对实际生产工程的指导意义正在边际递减。行业现在真正需要的是基于自身代码仓库（Repo）、业务特征与工程团队习惯所建立的私有基准测试套件（Suite Benchmarks）。 — Chase Norton
> [11:40](https://www.youtube.com/watch?v=c-YRayt-SGM&t=700s) 工程师通常有一种职业本能——倾向于直接选择市面上最新、最大、最快的模型，想要亲自体验新技术并在脑海中建立对新模型的认知。然而，通过将真实的工程任务、代码库和历史需求投入基准测试后，Faros 团队发现直觉往往并不可靠。 — Chase Norton
> [11:40](https://www.youtube.com/watch?v=c-YRayt-SGM&t=700s) 在排行榜上名列前茅的往往并不是 Claude Opus 这类顶级巨无霸模型，一些轻量或特定适配的模型在特定任务中的表现和对齐度反而更高。这证明了脱离自身业务场景去盲目追捧大模型很容易造成资源浪费 — Chase Norton
> [13:04](https://www.youtube.com/watch?v=c-YRayt-SGM&t=784s) 这种优化不再局限于个别工程师的微调，而是直接赋能并改变整个工程组织的 AI 使用方式，形成一个“观察、仿真、审查、治理”（Observe, Simulate, Review, Govern）的组织级自学习闭环。 — Chase Norton
> [15:09](https://www.youtube.com/watch?v=c-YRayt-SGM&t=909s) 实现 AI 价值最大化（Outcome Maxing）的核心在于三步走：首先捕获完整交互日志，其次将其深度连接到软件开发生命周期（SDLC）中以主动挖掘低效与浪费，最后通过动态网关进行策略路由与持续调优。 — Chase Norton
## Mentioned
Chase Norton, Faros, Ai4, Whiplash Report, Faros 2026 AI Engineering Report, OpenTelemetry (OTel), Claude Code, Claude Opus, Fable, Kimi K3, GLM, GitHub, GitLab, Jira, Linear, Faros AI Gateway, index.faros.ai
## Video description
Chase Norton - Head of Artificial Intelligence, Faros
AI adoption is no longer the hard part; proving what it changed is. Engineering teams are now running AI-native software factories where tokens, agents, prompts, pull requests, incidents, reviews, and rework all move through different systems, while finance is asked to approve bigger AI budgets with little evidence of return. In this talk, I will show how to evaluate AI investment through work evidence: which AI activity reached shipped software, which work created leverage, and which spend became rework, overhead, or waste. We will use patterns from Faros' 2026 AI Engineering Report, based on two years of data from 22,000 developers and 4,000+ teams, to explain why adoption and throughput are not the same as ROI. Attendees will leave with a practical framework for tying AI dollars to shipped work, setting ROI guardrails, and continuously monitoring whether AI is creating capital or compounding waste. If you are accountable for AI investment and need a sharper answer than "we bought more tools," this session is built for you.
[Source](https://www.youtube.com/watch?v=c-YRayt-SGM)
[0:04] I am from Hawaii. I I commute back and forth
from Hawaii to San Francisco. And I am just
[0:11] never surrounded by this. And you know, it's been
about what four or five years now that we've kind
[0:18] of had this alien technology come into us and us
trying to wield it and use it. And I think we're
[0:23] all learning individually how to master it, how
to get outcomes from it, how to make it do what
[0:30] we hope it will do. Throughout this talk, uh, I'm
cha I'm Chase Norton with Ferros and we're going
[0:36] to dive into a framework that we've built to help
us to try to understand not just token maxing,
[0:43] which I think was really like 2023, 2024, but
now moving into the time where we have to produce
[0:49] uh we have to give ROI. We have to actually
make this produce outcomes or justify them. So,
[0:55] how do we get to that stage? I am head of AI here
at Ferros and so everything that I go through is
[1:03] I am hands-on. This is the implementation that
we have done. This is kind of into the weeds
[1:09] of it a little bit. So we're going to get a
little technical but also something that you
[1:13] can take away and at the very very least have a
framework for how to think about how can we start
[1:18] to actually capture or define outcomes. How can
we start to give things back to the board but also
[1:24] down to the engineers. So bottom up and top down.
A year ago, we produced we we did a study across
[1:32] 22,000 engineers over two years that looked at how
is the AI spend or AI usage affecting PRs. Uh this
[1:41] is called the whiplash report. Welcome to look it
up. What we saw is we have about a 51% increase in
[1:48] uh PR sizes. But mo most importantly, this number
right here, it's taken about five times longer to
[1:54] review the PRs. We've we've moved the bottleneck
down to now we can produce code really quickly,
[2:01] but engineers are now forced to review it or,
you know, now teams are starting to look at how
[2:05] can we automate that? How can we have AI review
the PRs? That gets controversial and you'll have
[2:11] heated debates about that, I'm sure. But what we
do see is the bottleneck has now shifted. we can
[2:16] produce code fast. Uh but we now have to review
it. And the the question really comes is even
[2:22] though we can produce it fast, even though we can
burn these tokens and as the harnesses get to the
[2:28] point now that they can spin up an agent swarm of
a hundred agents at once and burn through all of
[2:32] your token spend, is it actually producing things
that the company cares about? Is it producing uh
[2:40] anything that we can go and say like this was our
result and this is how we are using our AI usage.
[2:46] So looking first unit of optimization kind of what
we're going to go through in this session or in
[2:51] this talk is looking at task. So you can think
about it there's three components to this that
[2:57] uh you can kind of form in your head which is you
have strategic initiatives what the company cares
[3:02] about. This is defined normally in Jira, linear,
some kind of project management tool. These have
[3:08] epics. These have tasks the individual engineer
is using. We can also group the epics by what does
[3:14] leadership care about? It's not just the single
solo uh epic. We also have engineering artifacts.
[3:22] What is the engineer producing? What PRs in
GitLab, GitHub, where do these live? We also have
[3:28] the AI session. this session you can think about
everything else I've mentioned has been a part of
[3:33] the normal SDLC that we know for many many years
forever uh and we have great methods to analyze
[3:40] that and model the SCLC but now we have something
called AI sessions this thing that even though it
[3:47] is a part of the SCLC it's kind of this like it
lives in its own space and really it shouldn't
[3:53] and so throughout all of this it's how do we bring
it into our normal software development life cycle
[3:58] graph in In order to tell the richness, the story
there that we've been able to tell with every
[4:04] other component, everything starts with telemetry
or in this case, however you want to collect your
[4:13] sessions. But the lowest hanging fruit, if you
want to walk away right now, is just do this. Make
[4:19] sure that in some way your company is collecting
your session logs, whether or not you're using
[4:23] them right now, being able to collect them the
easiest way. again lowest hanging fruit just an
[4:29] hotel collector that you point to claude codeex
they all have this enabled so and as a first
[4:36] class citizen in their harnesses where they will
batch it out for you they'll send you rich metrics
[4:40] or or metadata about it and so this is the first
how can we collect our own and I would say it's
[4:48] not sufficient enough to just go into the cloud uh
dashboard and look at what is your token spend how
[4:55] many tokens have you used what is the leaderboard.
These are metrics that we've been looking at for
[5:00] years now and they tell a part of the story, but
there is so much more in those logs. Whether or
[5:06] not it's the traces, the metrics, the logs,
whichever endpoint you're you're capturing,
[5:11] this is one of the lowest hanging fruits that
most people I see are not doing right now. That
[5:15] just enabling this allows you to capture uh PRs,
branches, commit shaws. All of this lies inside
[5:24] of those logs. You can ena enable prompts to
be uh sent. You can enable response bodies,
[5:31] tools. All of that is in there and all of that is
there for you to analyze once you capture it so
[5:37] that you can begin to tell your own story whether
or not it's in cloud codecs across multiple
[5:43] different providers. They all support this uh
first class. They also there is a very easy
[5:48] open-source hotel collector. Uh so you can just
spin that up and point everything to that and you
[5:54] can get going by the end of the night. The reason
for this is first we start with can we actually
[5:59] capture the sessions? Can we actually capture
these metrics? Again the goal is can we pull
[6:04] this into the overall traditional SDLC graph that
we know in order to tell that story once we can
[6:12] and you can actually see here this is part of like
Ferros on Feroh spend here. uh if we can then it's
[6:20] not just like the spend we also want to be able
to tell can we attribute these sessions to work
[6:26] to engineering artifacts to task as I mentioned
before once we have those logs kind of gave the
[6:34] spoiler away earlier but inside of those logs
is a lot of different signals that allow us to
[6:41] say this session or this event because most hotel
is uh sent out in events Because sessions can we
[6:48] all do AI differently. Sessions can be multi-day
sessions. If you're some people, some people do a
[6:53] single session per per uh task. But they're sent
out as an events and in those events are attached
[7:00] different signals that allow us to say yes, we can
actually connect this to APR. We can connect this
[7:06] to an engineer. We can connect this to a commit.
Being able to attribute in this way, being able to
[7:14] pull off commit shaws that exist, the branches
that were used, all of that data allows us to
[7:19] go from all of a sudden it the AI sessions live
outside to now we're starting to bring them inside
[7:25] of our SDLC graph. But it's not just GitHub. It's
also we can connect this once they're inside of
[7:35] uh once we've made the attribution to GitHub, then
it just becomes normal processing. We can connect
[7:39] GitHub to Jira just like we've always done, right?
Once we connect it to Jira, you can see the story.
[7:47] The story starts getting told. We have session
data tied to engineering artifacts that are tied
[7:54] to strategic initiatives. These tasks are then can
be reviewed to see was the PR closed? Was the task
[8:02] actually completed? Does this epic live inside
of one that leadership cares about? And if so,
[8:09] how are they grouped together? Once you can
connect that story, it becomes something of
[8:14] a modeling problem. And honestly, I didn't know
I was going to be in a room of just all AI ROI,
[8:19] but that's awesome because it means everyone
here cares about that. But I've watched a lot
[8:23] of the other talks and I think there's almost
like a blend of taking some of the things I'm
[8:27] suggesting here along with some of the things
other people have been suggesting. How do you
[8:31] actually model it once you get to that level? How
do you how do you create the the algorithm and
[8:36] the and the the modeling for your own ROI? That
is up to you. I can tell you I've been in those
[8:42] meetings. Everyone wants to do it differently. So,
that's not something that I'm going to solve. But
[8:46] the point is is allow yourself to get to that
attribution state where you can say with some
[8:52] level of confidence that the uh AI sessions
are connected throughout that story chain.
[9:03] The incredible thing though is is once you make
that connection, so again now like now we're no
[9:08] longer on the outside. We're now inside. We
now have our AI sessions a part of everything
[9:14] else. And you know all the connections, the
attribution, but you can now tell something.
[9:19] You can now say to yourself, I know exactly which
model, all the configurations of this AI session
[9:25] that went into actually doing this task, doing
this exact work, this PR. And if you can do that,
[9:33] then you can actually roll back time and you can
say, "What if I change something?" And you can
[9:38] start to build what if scenarios. You can start to
say to yourself, "What if I change the model, the
[9:43] harness, the effort? What about the context? what
about the agents.mmd file? And all of a sudden,
[9:50] it becomes your world to play with of how you're
going to model or how you're going to build these
[9:55] what if scenarios. And so, just the fact that
you've been able to now make that connection
[10:00] allows you to say, well, what if my engineers
are always using Fable? Do they do they have to
[10:09] can they actually be be running it on 5.6 Terra to
save a little bit more money or Luna now that it's
[10:15] 80% off? But do they actually need to be using
that? Is there a more efficient way to run to have
[10:24] my engineers running their AI? Million different
what if scenarios that you can create. But this
[10:30] allows for that connection in the beginning of
those what if scenarios that you can hypothesize
[10:34] about even to the point that I would say like it's
not that I'm done with benchmarks but there's just
[10:42] so many benchmarks that at some point it just
becomes like okay what is this one um and so I
[10:49] think we're at a point now where we can tell this
story we can make the connections we can make the
[10:53] attribution and I think we're finally at a point
where we can build our own suite benchmarks And I
[10:59] think that that's more relevant at least to me as
we're looking at Kimmy K3 and we're looking at GLM
[11:05] 5.2. I forget if 5.3 got got released, but we're
looking at openweight models versus Opus versus
[11:11] uh 5.6 and we're trying to say what is best for
us, what is best for Ferros, what is best for our
[11:18] repo, what is best for this team. And sometimes
those are different. And I would say often they're
[11:24] different from what we see in the more generic
benchmarks. But because we can actually start
[11:29] to do our own what if scenarios, we can start to
say, what if we change the harness? What if we
[11:34] change the model? What is the best configuration
that really aligns with the way that we are using
[11:40] AI or the way we do work in this repo versus
this one? So the idea is it's not just about
[11:47] outcomes. Outcomes become something that once
you can tell the story, you can now enhance. You
[11:53] can now not just do outcomes, but you can also do
your own whatifs. and you can start to build your
[11:58] own company benchmark. And so this is where for
instance inside of Ferros, if you're interested,
[12:06] check out index.faros.ai. This is where we've done
this ourselves, right? And we we we put it out
[12:11] there so that you can see we've run it on our own
repos, our own codebase, our own uh our own task,
[12:18] generated those, and we've been able to say,
guess what? It's not Opus. It's not even
[12:23] um it's not even 5.6. that is really topping the
the leaderboard for us. GLM 5.2 aligns a little
[12:32] bit more. Open code harness is is higher up on the
leaderboard. But you can check it out where it's
[12:38] not just the most uh to us at least as an engineer
sometimes you just grab for the best the fastest
[12:45] because you want to also you want to experience
it. You want to be a part of like whatever the new
[12:49] is so that you can start to build your mental
model of is this actually good? And it turns
[12:55] out that if we benchmark it for our repos and our
work, it's not always what my gut intuition says.
[13:04] The last point to this is it's not just about
so so hopefully there's been a story here that
[13:11] you can tell or you you can uh understand of
having the sessions connected throughout this
[13:16] whole chain. We can do the whatifs. We can do the
benchmarking. But what if you learn that along the
[13:25] way you actually can discover inefficiencies
where you can see patterns? You can see where
[13:31] some people are doing X and some people are
doing Y, but people who do X are producing more
[13:37] efficient sessions, producing more outcomes. In
that case, what you can start to do is actually
[13:43] build policies, right? So you can start to say
when we see this route here, we see this route
[13:50] here. And I say route because I I think I'm
giving it away a little bit. But there also
[13:56] if you combine all of these techniques along with
your own or Ferris's gateway and point it to it,
[14:04] you now have absolute control. So point all AI
through one gateway. Apply policies, policydriven
[14:12] routing to this gateway. And then as you are
running these whatifs, as you batch off events
[14:19] and you start to learn what is inefficient, what
is productive, where are engineers succeeding,
[14:24] where are they being redundant, you can modify
those policies, apply the policy config at the
[14:30] gateway, change and optimize not just in a single
engineer, not just AI, like a single AI usage,
[14:38] but the way that the entire engineering org uses
AI and it impacts every single person and then as
[14:45] that config changes you learn more or rather the
system learns more. You can kind of see what's
[14:51] going on here. It becomes a self-learning loop
of optimizing for your entire engineering org.
[15:01] That's what we say right here for the Ferros AI
gateway. Observe, simulate, review, and govern.
[15:09] I think throughout this throughout this talk
there's I I instead of showing you what our
[15:15] product is though I do welcome you please go to
ferris.ai and check it out. We're all trying to
[15:20] solve this together and and I don't think there's
ever been such a time where there has not been a
[15:26] path that you know to get on to go to get to the
end result right used to be if you wanted to be
[15:33] a great engineer you did a certain path you
want to be a great leader you went a certain
[15:38] path but with AI we are all on our own on our
computers figuring this out and I think the
[15:43] more that we can kind of share at at the very
least get into a framework capture the logs
[15:50] from from being able to capture the logs, connect
them into your SDLC, from connecting into the
[15:56] SDLC. Start asking questions, start looking for
inefficiencies. You'll find them. 100% guarantee
[16:01] you'll find them. Uh, and they're not that hard
to find. Making the connection. And then finally,
[16:10] if you are willing to take it to the next level
of being able to route everything through a way
[16:15] that you can dynamically change based on how you
are improving and where your inefficiencies are.
[16:22] This is the way that we've been able to find
uh what we call outcome maxing at this point,
[16:26] but at least maximizing outcomes, being able
to communicate ROI, being able to communicate
[16:31] a pattern in a way for leadership to understand
and for engineers to see transparently what is
[16:37] going on and for them to improve themselves. So,
with that, uh, this is kind of the the three-step
[16:46] system at least that that I wanted to share,
but I'll be around for any questions, anything
[16:51] that you may have. Uh, again, check it out at
index.far.ai or ferris.ai AI or just come talk to
