Shopify Gives Engineers Unlimited AI Tokens Under Barbell Strategy

Claude

Shopify CTO Mikhail Parakhin warns rival tech executives make a critical strategic error by capping developer tokens while wasting capital on unoptimized production models.

AI-assisted summary of Inside Shopify's AI strategy with CTO Mikhail Parakhin — original source. Check the original for full context.

Source published: 2026-10-08

Key ideas

Shopify CTO Mikhail Parakhin advocates an AI 'barbell strategy': granting engineers unlimited access to the largest frontier models during development, while aggressively optimizing and distilling smaller models for production to reinvest the savings. (9:41)
Shopify created a predictive 'digital twin' of merchant businesses by modeling organizational actions as sequential tokens, enabling counterfactual simulations across credit, ad campaigns, and shipping transit times. (4:42)
Parakhin argues that the true leverage of AI lies in 'raising the ceiling'—solving previously intractable problems via a human-in-the-loop 'centaur' approach—rather than merely automating routine toil to 'raise the floor.' (0:35)
Shopify tracks engineering ROI by normalizing project velocity against algorithmic estimates of problem complexity, validating statistically significant double-digit productivity gains. (2:07)

Detailed notes

## Background Published on YouTube by the official Claude channel on October 8, 2026, this video features a conversation between host Boris and Shopify CTO Mikhail Parakhin. The discussion covers how Shopify leverages Claude across its engineering organization, focusing on giving engineers unlimited tokens with frontier models, constructing an enterprise digital twin, and executing a distinct AI resource strategy.

## TL;DR Shopify CTO Mikhail Parakhin explains that the true power of frontier AI lies in expanding intellectual capability to solve previously intractable problems rather than simply automating routine tasks. He outlines how Shopify treats business events as token sequences to create digital twins that simulate merchant growth scenarios, enabling proactive, risk-free business interventions. To maximize engineering velocity, Parakhin champions a barbell token strategy that grants developers uncapped access to the largest frontier models while aggressively optimizing smaller, fine-tuned models in production. He also highlights how AI telemetry aids team leadership and wryly notes how rapid AI progress recently beat him to his retirement goal of deciphering ancient Linear A script.

## Atmosphere & style The conversation is pragmatic, high-velocity, and intellectually rigorous, driven by Parakhin's deep technical conviction and executive clarity. The dynamic between Boris and Parakhin feels collegial and candid, moving smoothly from serious discussions on mathematical optimization and systems architecture to shared laughter over AI encroaching on personal intellectual pursuits. It conveys the electric, fast-moving reality of frontier AI deployment inside a major tech company without devolving into vague hype.

## Bulletin - [09:41](https://www.youtube.com/watch?v=9GtcbnmldcY&t=581s) Shopify CTO Mikhail Parakhin advocates an AI 'barbell strategy': granting engineers unlimited access to the largest frontier models during development, while aggressively optimizing and distilling smaller models for production to reinvest the savings. - [04:42](https://www.youtube.com/watch?v=9GtcbnmldcY&t=282s) Shopify created a predictive 'digital twin' of merchant businesses by modeling organizational actions as sequential tokens, enabling counterfactual simulations across credit, ad campaigns, and shipping transit times. - [00:35](https://www.youtube.com/watch?v=9GtcbnmldcY&t=35s) Parakhin argues that the true leverage of AI lies in 'raising the ceiling'—solving previously intractable problems via a human-in-the-loop 'centaur' approach—rather than merely automating routine toil to 'raise the floor.' - [02:07](https://www.youtube.com/watch?v=9GtcbnmldcY&t=127s) Shopify tracks engineering ROI by normalizing project velocity against algorithmic estimates of problem complexity, validating statistically significant double-digit productivity gains.

## Key takeaways - [00:00](https://www.youtube.com/watch?v=9GtcbnmldcY&t=0s) Enterprises can be treated as sequence prediction systems, representing continuous operational events (credit, inventory, site changes) as tokens to build a digital twin. - [00:35](https://www.youtube.com/watch?v=9GtcbnmldcY&t=35s) The primary strategic value of frontier models is 'raising the ceiling' to solve problems out of reach of human intuition, rather than solely 'raising the floor' via automation. - [00:35](https://www.youtube.com/watch?v=9GtcbnmldcY&t=35s) Human-model 'centaur' workflows outperform both solo human experts and standalone base frontier models when tackling complex mathematical and technical challenges. - [02:07](https://www.youtube.com/watch?v=9GtcbnmldcY&t=127s) Measuring engineering ROI requires normalizing output against intrinsic project complexity rather than relying purely on raw commit or PR volume. - [03:32](https://www.youtube.com/watch?v=9GtcbnmldcY&t=212s) Shopify provides engineers with unlimited tokens and immediate access to the largest models to prevent artificial friction from hiding the true capability frontier. - [03:32](https://www.youtube.com/watch?v=9GtcbnmldcY&t=212s) Technology leadership and budget approvers must use frontier models daily to genuinely understand and greenlight capability unlocks. - [04:42](https://www.youtube.com/watch?v=9GtcbnmldcY&t=282s) Shopify's digital twin allows risk-free counterfactual simulations of merchant decisions, such as marketing tests, loan allocations, and shipping transit adjustments. - [04:42](https://www.youtube.com/watch?v=9GtcbnmldcY&t=282s) Simulated operational pathways directly trigger real-world interventions and prescriptive recommendations, driving merchant success and company revenue. - [07:28](https://www.youtube.com/watch?v=9GtcbnmldcY&t=448s) AI operational telemetry helps engineering leadership spot mathematically drifting deadlines and cross-team friction before morale degrades. - [07:28](https://www.youtube.com/watch?v=9GtcbnmldcY&t=448s) Engineers are evolving into 'second-level managers' whose core role is orchestrating, guiding, and resolving conflicts across autonomous agent workflows. - [07:28](https://www.youtube.com/watch?v=9GtcbnmldcY&t=448s) Model sycophancy poses a new technical management risk, as flawed prompts quickly yield bad code without traditional team debates serving as a buffer. - [09:41](https://www.youtube.com/watch?v=9GtcbnmldcY&t=581s) The recommended 'barbell token strategy' pairs unconstrained frontier reasoning in development with fine-tuned, cost-efficient models in production. - [10:40](https://www.youtube.com/watch?v=9GtcbnmldcY&t=640s) Parakhin considers restricting internal developer tokens while running bloated, expensive models in production to be a critical strategic error. - [11:34](https://www.youtube.com/watch?v=9GtcbnmldcY&t=694s) Frontier models are progressing so rapidly that long-standing human intellectual challenges, like deciphering Linear A, are being solved before researchers can get to them.

## Detailed walkthrough ### [00:00](https://www.youtube.com/watch?v=9GtcbnmldcY&t=0s) The Concept of the Company Digital Twin

Just as natural language models learn to predict the next word or letter in a sequence to generate coherent text, human behavior and company actions follow sequential patterns. A company can be mathematically represented as an evolving sequence of operations: opening lines of credit, shipping inventory, running advertising campaigns, modifying web storefronts, or onboarding payment processors. By treating this continuous stream of organizational events as tokens in a sequence, it becomes possible to train models to construct a comprehensive "digital twin" of the enterprise.

```mermaid flowchart LR accTitle: Company Digital Twin Sequence Architecture accDescr: Mapping enterprise events into a sequence prediction pipeline to create a digital twin. subgraph Stream["Enterprise Event Stream"] A["Credit & Loans"] B["Catalog & Site Updates"] C["Ad Campaigns & Shipping"] end subgraph Model["Predictive Engine"] D["Sequence Model"] E["Digital Twin Simulator"] end subgraph Simulation["Outcomes"] F["Counterfactual Testing"] G["Optimized Growth Action"] end

A --> D B --> D C --> D D --> E E --> F F --> G ```

[00:35](https://www.youtube.com/watch?v=9GtcbnmldcY&t=35s) Raising the Ceiling vs. Raising the Floor

When exploring the limits of frontier models like Claude, the most significant breakthrough does not lie in eliminating everyday toil or automating tedious tasks ("raising the floor"). Instead, the transformative leverage comes from expanding what is intellectually reachable ("raising the ceiling"). Mikhail Parakhin shares a concrete personal example: for seven years, he wrestled unsuccessfully with an intractable mathematics and machine learning challenge involving the Wasserstein loss function ("making everything Gaussian"). Neither human intuition nor earlier LLMs had managed to solve it.

The breakthrough arrived through an interactive, human-in-the-loop dynamic—a "centaur" approach akin to freestyle chess. The human provides the steering, framing, and critique, while the model navigates computational and conceptual pathways that neither could master in isolation.

- Raising the floor only frees up time by automating work that could otherwise be accomplished with more headcount or greater diligence. - Raising the ceiling breaks open problems that would remain unsolved even if you had "thousands of the best mathematicians" at your disposal. - The hybrid "centaur" workflow outperforms the standalone capabilities of either the human expert or the base frontier model.

[02:07](https://www.youtube.com/watch?v=9GtcbnmldcY&t=127s) Measuring Productivity and Engineering ROI

Quantifying the return on investment for efficiency gains (raising the floor) is relatively simple across the industry: organizations measure pull requests (PRs), code commit frequencies, and engineering hours saved. Shopify approaches this with high rigor, tracking not just raw output volume or PR counts, but algorithmically estimating the intrinsic complexity of projects.

Using internal AI-driven execution trackers (Shopify's "Get Stuff Done" culture and tooling), leadership tracks project velocity normalized against problem complexity. This metric establishes with statistical confidence that engineering teams are delivering more complex initiatives at a significantly accelerated pace, proving quantifiable double-digit productivity gains.

[03:32](https://www.youtube.com/watch?v=9GtcbnmldcY&t=212s) The Philosophy of Unlimited Tokens and the Largest Models

When managing engineering investments and having discussions with leadership like Shopify CEO Tobi Lütke, the guiding policy is absolute commitment to scale. Parakhin advises other CTOs to resist the instinct to cut costs prematurely on reasoning and inference:

- **Always deploy the largest model first:** You cannot understand what capabilities you are leaving on the table unless you evaluate against the absolute state of the art. Smaller models conceal the true capability ceiling. - **Unlimited tokens across the engineering floor:** Shopify provides unrestricted access to model tokens so engineers can experiment without artificial friction. - **Heavy infrastructure backing:** The company commits massive GPU inference and training resources to ensure developers never face resource bottlenecks. - **Decision-maker immersion:** Leadership and budget approvers must actively use the frontier tools daily; understanding capability unlocks cannot be done via abstract presentations.

[04:42](https://www.youtube.com/watch?v=9GtcbnmldcY&t=282s) Building a Merchant Growth Engine via Digital Twins

Shopify translated the digital twin concept into production through its merchant prediction system. An enterprise is made up of millions of microscopic actions taken by staff, inventory systems, and customers every second. By encoding these discrete events into a unified historical sequence, Shopify constructs a functioning model of the merchant's business.

This allows Shopify to simulate counterfactual business decisions in a risk-free environment without gambling real-world capital or merchant survival: - Simulating the impact of running marketing campaigns across platforms like X (Twitter). - Modeling whether reducing shipping transit times by twenty-four hours tips a business into profitability. - Testing whether extending a tailored Shopify Capital loan will unlock an optimal sequence of business actions that guarantees growth.

Once an optimal path is identified within the simulation, Shopify proactively triggers real-world interventions—contacting the merchant with prescriptive recommendations or automated financial offerings. This capability has produced a measurable, direct impact on merchant success and Shopify's bottom line—a project that Parakhin notes "just wouldn't have been done" previously due to the sheer human talent and coordination required.

[07:28](https://www.youtube.com/watch?v=9GtcbnmldcY&t=448s) Transforming Engineering Leadership and Human-in-the-Loop Management

Parakhin previously believed language models were ill-suited for organizational management and team leadership, but reversed his stance after building an operational telemetry system. The system evaluates cross-team work patterns and flags risks before humans notice them: - Proactively alerting leadership when project delivery dates are mathematically drifting off course. - Identifying systemic friction points and team dissatisfaction before morale degrades into attrition.

At the same time, the technical demands placed on managers and senior engineers have intensified. Because models exhibit strong sycophancy—aiming to please by validating assumptions embedded within user prompts—a poorly articulated technical vision will immediately produce flawed implementations without the historical safety buffer of prolonged team debate.

Engineers are moving from typing code to acting as "second-level managers." Their primary responsibility is supervising autonomous agents, delegating and routing tasks effectively, maintaining global context across disparate agent streams, and resolving conflicting implementation patterns.

[09:41](https://www.youtube.com/watch?v=9GtcbnmldcY&t=581s) Advice for CTOs: The Barbell Strategy

For technical leaders scaling Claude across engineering organizations, the core mandate is to avoid artificial token throttling. Beyond having safety circuit breakers in place to kill runaway recursive loops, CTOs should avoid assuming smaller models are "good enough" for engineering workflows.

Parakhin outlines a strict **barbell token strategy**: - **Development, Coding, Architecture, and Research:** Commit the budget exclusively to the absolute largest, most capable frontier models available to maximize reasoning and capability. - **Production Execution:** Downscale, distill, or optimize for targeted, cost-effective inference once requirements and systems are finalized.

[10:40](https://www.youtube.com/watch?v=9GtcbnmldcY&t=640s) The Barbell Token Strategy: Cheaper Production, Unlimited Development

Mikhail Parakhin breaks down the operational philosophy behind Shopify's AI resource allocation, describing what amounts to an aggressive "barbell strategy." On the production side, the mandate is to ruthlessly optimize: fine-tune smaller, cheaper models and "auto-research the hell out of it" to strike the exact right trade-off between inference cost, throughput, and output quality. The substantial capital saved by running lean, highly tailored models in production is then redirected straight into internal development, where engineers are given maximum horsepower and the largest, most capable frontier models available. When a frontier jump occurs, the leap in engineering capability is monumental—a step change that Parakhin notes was on record as being vastly bigger than any prior generational shift.

He observes with bewilderment that many tech executives end up doing the exact opposite. Companies will pinch pennies on internal engineering workflows by restricting coding tokens, only to bleed money in live environments by naively routing production traffic through massive, oversized generalist models where they are not needed. For Parakhin, this inverted approach is a massive strategic blunder: "I feel like I will outcompete anybody who is not on that strategy."

```mermaid flowchart LR accTitle: The Barbell Token Strategy accDescr: Diagram illustrating Shopify's barbell token strategy balancing development and production.

subgraph Dev["Internal Development"] D1["Frontier LLMs"] --> D2["Unlimited Tokens"] D2 --> D3["Maximum Capability & Speed"] end

subgraph Prod["Production Services"] P1["Fine-Tuned Smaller Models"] --> P2["Auto-Researched Trade-offs"] P2 --> P3["Optimized Cost & Throughput"] end

Prod -.->|"Cost Savings Reinvested"| Dev ```

- **Production Optimization:** Use targeted, fine-tuned, and cost-effective models; systematically evaluate and benchmark throughput, latency, and quality. - **Development Investment:** Fund cutting-edge frontier model access without token caps for engineers, capturing the massive potential of architectural capability leaps. - **Common Industry Anti-Pattern:** Artificially throttling developer tokens while wastefully running expensive, oversized frontier models on repetitive production workloads.

---

[11:34](https://www.youtube.com/watch?v=9GtcbnmldcY&t=694s) The Stolen Retirement Project: Deciphering Linear A

The conversation takes a humorous, personal turn as Parakhin jokes about how Claude and host Boris ruined his lifelong passion project. Parakhin recounts how he would frequently tell friends that his ultimate retirement plan was to crack Linear A—the famous, still-undeciphered Bronze Age writing system used by the Minoan civilization on the island of Crete. For years, deciphering the ancient script had been his sacred personal intellectual pursuit, something he planned to finally conquer later in life, perhaps tinkering away on an old Commodore computer.

That dream was shattered when Boris posted a tweet showcasing how an AI researcher managed to make significant progress deciphering Linear A using Claude. Parakhin laments the moment with exasperated amusement: "Damn it, that was my whole spiel. Took my retirement project... I just didn't finish in time." Boris and Parakhin laugh over how rapidly frontier models are claiming even the most obscure, long-standing human puzzles before people can get around to working on them.

## Notable quotes > [00:35](https://www.youtube.com/watch?v=9GtcbnmldcY&t=35s) making everything Gaussian — Mikhail Parakhin

> [00:35](https://www.youtube.com/watch?v=9GtcbnmldcY&t=35s) raising the floor — Mikhail Parakhin

> [00:35](https://www.youtube.com/watch?v=9GtcbnmldcY&t=35s) raising the ceiling — Mikhail Parakhin

> [00:35](https://www.youtube.com/watch?v=9GtcbnmldcY&t=35s) thousands of the best mathematicians — Mikhail Parakhin

> [04:42](https://www.youtube.com/watch?v=9GtcbnmldcY&t=282s) just wouldn't have been done — Mikhail Parakhin

> [07:28](https://www.youtube.com/watch?v=9GtcbnmldcY&t=448s) second-level managers — Mikhail Parakhin

> [09:41](https://www.youtube.com/watch?v=9GtcbnmldcY&t=581s) barbell token strategy — Mikhail Parakhin

> [10:40](https://www.youtube.com/watch?v=9GtcbnmldcY&t=640s) auto-research the hell out of it — Mikhail Parakhin

> [10:40](https://www.youtube.com/watch?v=9GtcbnmldcY&t=640s) I feel like I will outcompete anybody who is not on that strategy. — Mikhail Parakhin

> [11:34](https://www.youtube.com/watch?v=9GtcbnmldcY&t=694s) Damn it, that was my whole spiel. Took my retirement project... I just didn't finish in time. — Mikhail Parakhin

## Mentioned Boris, Claude, Claude Code, Commodore, Linear A, Mikhail Parakhin, Shopify, Shopify Capital, Tobi Lütke, Wasserstein loss function, X (Twitter)

## Video description Shopify's CTO on how Shopify uses Claude: unlimited tokens for everyone, always the largest model, and a digital twin of the company.

In this conversation with Boris, Shopify's CTO explains why the real value of working with Claude is in building things that weren't possible before. He describes solving a problem he'd worked on for seven years, how Shopify built a digital twin of the company, and the unique barbell strategy he recommends for getting the most out of Claude.

Have a question? Let us know in the comments.

Learn more about Claude Code: https://claude.com/claude-code

Chapters 0:00 How Shopify's CTO raises the ceiling with Claude 2:07 Measuring AI ROI and engineering productivity 3:32 Why Shopify gives every engineer unlimited tokens 4:42 Building a digital twin to predict merchant growth 7:28 AI for managing engineering teams 9:41 Advice for CTOs: the barbell token strategy

[Source](https://www.youtube.com/watch?v=9GtcbnmldcY)

中文简报

Shopify推行“杠铃策略”，为工程师提供无限制AI Token Shopify CTO Mikhail Parakhin警告称，竞争对手的高管限制开发者的Token额度，却在未经优化的生产模型上浪费资金，犯下了严重的战略错误。 Shopify CTO Mikhail Parakhin解释称，前沿AI的真正威力在于拓展智力能力以解决以往棘手的难题，而非仅仅自动化常规任务。他阐述了Shopify如何将商业事件视为Token序列来构建数字孪生，从而模拟商家的增长场景，实现主动且无风险的业务干预。为了最大限度提高工程研发效率，Parakhin倡导一种杠铃式Token策略：在开发端允许工程师无限制调用最大的前沿模型，同时在生产端大力优化经过微调的小型模型。他还强调了AI遥测技术对团队领导力的辅助作用，并诙谐地提及AI的飞速进展最近甚至抢先一步实现了他原本计划退休后去完成的目标——破解古老的线形文字A（Linear A）。 Shopify CTO Mikhail Parakhin倡导一种AI“杠铃策略”：在开发阶段让工程师无限制使用最大的前沿模型，而在生产环境中大力优化和蒸馏较小模型，以便将节省下的成本进行再投资。 (9:41) Shopify通过将组织行为建模为序列Token，为商家业务创建了预测性的“数字孪生”，从而能够在信贷、广告活动和物流运输时效等方面实现反事实模拟。 (4:42) Parakhin认为，AI的真正杠杆作用在于“提高上限”——即通过人机协同的“半人马”（centaur）模式解决以往难以攻克的难题，而不仅仅是通过自动化日常琐碎任务来“提高下限”。 (0:35) Shopify通过参照算法对问题复杂度的评估来将项目研发速度标准化，以此追踪工程ROI，验证了具有统计显著性的两位数生产力提升。 (2:07)

Reading summary

Shopify CTO Mikhail Parakhin argues that frontier AI creates the greatest enterprise value when paired with human expertise to expand intellectual capabilities and simulate business operations, provided leadership funds unrestricted development compute while optimizing production inference. Expanding Intellectual Reach via Centaur Workflows (0:35) Parakhin contends that the primary breakthrough of frontier models is "raising the ceiling" on what is intellectually achievable, rather than merely "raising the floor" by automating routine labor. While floor-level automation frees up hours that could otherwise be handled by additional headcount, ceiling expansion allows organizations to crack problems that remain intractable even to large teams of specialists. Parakhin points to his own experience resolving a seven-year mathematical challenge involving the Wasserstein loss function, accomplished through a collaborative "centaur" workflow combining human steering and critique with frontier model computation. Simulating Business Decisions with Enterprise Digital Twins (4:42) Shopify models organizations by treating discrete operational events—such as inventory shipments, ad campaigns, catalog edits, and credit lines—as sequential tokens analogous to text. By training models on this historical event stream, Shopify constructs operational "digital twins" to run counterfactual simulations without putting merchant capital at risk. These simulations test whether specific adjustments, such as reducing transit times by twenty-four hours, advertising on platforms like X, or securing customized Shopify Capital loans, will yield sustained profitability. When an optimal trajectory is identified, Shopify triggers proactive real-world recommendations and automated financial offerings, which Parakhin reports has measurably improved merchant outcomes and Shopify's revenue. The Barbell Token Allocation Strategy (9:41) To balance capability and infrastructure expenses, Parakhin advises engineering leaders to adopt a strict "barbell strategy." In internal research, architecture, and coding, teams should deploy the largest available frontier models with unrestricted token access, ensuring engineers capture major generational capability gains without artificial bottlenecks. In contrast, production systems should be aggressively optimized by fine-tuning smaller, cheaper models to find the precise balance between latency, throughput, and quality. Parakhin criticizes the common corporate pattern of rationing tokens for internal developers while paying for oversized frontier models to serve repetitive production traffic, arguing that capital saved in production should instead fund developer exploration. Complexity-Adjusted Metrics and Agent Supervision (2:07) To track AI-driven engineering gains, Shopify normalizes team velocity against algorithmic estimates of problem complexity rather than tracking raw metrics like pull requests or commit volume. According to Parakhin, this telemetry demonstrates statistically verified double-digit productivity gains on complex deliverables. This shift also changes internal organizational dynamics. Shopify uses telemetry to detect off-track project timelines and team friction points before they harm morale. Meanwhile, engineers increasingly act as supervisors overseeing autonomous agents. Parakhin cautions that because models exhibit sycophancy by validating user assumptions, ambiguous technical prompts can instantly generate flawed systems without the traditional buffer of peer review debate. While Parakhin positions unrestricted frontier model access and digital twin simulations as decisive competitive advantages, the operational approach shifts greater cognitive burden onto senior engineers, requiring rigorous prompt precision to prevent model sycophancy and circuit breakers against uncontrolled autonomous loops.

Source quotes

[0:35] Mikhail Parakhin: making everything Gaussian [0:35] Mikhail Parakhin: raising the floor [0:35] Mikhail Parakhin: raising the ceiling [0:35] Mikhail Parakhin: thousands of the best mathematicians [4:42] Mikhail Parakhin: just wouldn't have been done [7:28] Mikhail Parakhin: second-level managers [9:41] Mikhail Parakhin: barbell token strategy [10:40] Mikhail Parakhin: auto-research the hell out of it [10:40] Mikhail Parakhin: I feel like I will outcompete anybody who is not on that strategy. [11:34] Mikhail Parakhin: Damn it, that was my whole spiel. Took my retirement project... I just didn't finish in time.

Original transcript

[0:00] You know how text is just a sequence of letters? [0:02] And LLMs [0:04] learned really well how to predict the next, [0:06] and then they can complete the text. [0:08] Human actions are very similar, [0:09] but in our case, more importantly, companies [0:12] actions are similar. [0:14] You can represent a company as a sequence of actions. [0:17] And then what happens is that you can create [0:19] a digital twin of the company. [0:35] What are the hardest problems that [0:38] you're throwing at Fable? [0:38] That you're throwing a Claude? [0:40] Where you've been surprised by success? [0:42] I came up with a solution to a problem [0:44] that for seven years I've been trying to solve. [0:47] And I was unable to solve it, and LLMs were not able to solve it. [0:51] Something called the Wasserstein loss function. [0:54] Making everything Gaussian. [0:56] It's not that [0:58] LLMs take away the drudgery of work [1:01] or the work that you didn’t want to do. [1:03] And now they do it. [1:04] And so now you can do interesting things, [1:07] more sophisticated. [1:10] It’s that you can do things that you previously couldn't. [1:13] Like, in no amount of time, no amount of helpers, [1:16] I could have had thousands of the [1:17] best mathematicians at my disposal, [1:19] I still wouldn't be able to do it. [1:21] But with the model I can. [1:24] And for the best problems right now what I see is that [1:27] I cannot solve it without the model, [1:29] but the model cannot solve it [1:30] without me either. [1:32] It requires, this back and forth and [1:35] creating that centaur. [1:37] You know, using chess analogy, [1:39] that together is better than the model [1:43] and better than a human being. [1:45] So you're raising the floor. But actually, don't forget, [1:47] you can also raise the ceiling. [1:48] Exactly. That's that is my point. Yeah. [1:50] Raising the ceiling is [1:54] is much more impactful. [1:56] Because raising the floor, [1:57] you could have just been more diligent [2:00] or throw more people at it. [2:01] Ceiling is, there's no easy way to extend. [2:04] Right. That's a real unlock. [2:06] So would you do this? I guess, for, [2:09] you know, like, [2:10] research or some problem that hasn't been solved before? [2:14] You can think about it that way. [2:15] This is a novel problem. [2:17] The solution was out of reach. [2:18] Now we've solved it. [2:20] In a company context, [2:21] in a business product context, how do you think [2:23] about the ROI or something like this? [2:25] Because something I hear a lot [2:26] is when you raise the floor, it's really easy to measure [2:29] because you're like, okay, this amount of work [2:31] would have been done by engineers. [2:32] Now, Claude can do it. Engineers are freed up. [2:35] That's fairly easy to quantify. [2:37] How do you think about the other kind of problem [2:38] where you've raised the capability ceiling [2:40] and now engineers are doing [2:41] something they couldn't have before? [2:43] It is a very good question. [2:45] And I feel like in terms of AI, [2:50] I'm just going to go ahead and say it, [2:51] it’s probably the best in the world right now at measuring [2:54] how high is the floor? [2:56] They have not only, of course, [3:00] just like everybody else, estimate the number [3:04] of PRs and complexity of PRs, [3:06] but we, we actually do things like estimating [3:09] complexity of the project. [3:11] And in our [3:13] AI system, just the system get stuff done. [3:16] We could we could see that, hey, the teams of people [3:19] are able to execute on more [3:23] projects faster and, normalized for complexity. [3:26] You can be very precise at measuring [3:29] percentages of productivity you're getting. [3:32] How do you have this conversation [3:34] with, with Toby and, [3:35] what's your advice to, another CTO [3:37] at a different company [3:39] that has to have the conversation [3:40] with their CEO about this? [3:41] always use the largest model [3:44] because the largest model [3:46] raises the ceiling the most first of all, [3:47] you don't know unless you use the large model [3:50] and then and compare the results. [3:51] You don't know what you're missing, right? [3:53] It's a no no, no. [3:55] And that and that they think stay in second. [3:57] Always throw maximum a thing at the problem. [4:01] And then until I see the evidence. [4:04] Otherwise that's what we're going to be doing. [4:06] And that's why, at Shopify [4:08] we have unlimited tokens, for everybody. [4:11] That's why run, the heaviest models. [4:14] We can find, that's why we, we throw a lot of, [4:17] a lot of resources in our with, GPU [4:20] inference and training and ability to optimize. [4:23] In a way, it's a it's, you just have to [4:26] you have to make sure that everyone that's making [4:28] the decisions is actually using the model [4:29] and using the product. [4:31] They understand what this is. [4:33] And then you have to just keep [4:35] pushing the ceiling and seeing those returns. [4:37] And then it's just obvious [4:38] if it works or it doesn't work. [4:39] What's an example of one of these projects [4:41] you ship that that raised the ceiling in an obvious way. [4:44] My the one that I'm probably, [4:47] most proud of is is our HST prediction system. [4:51] The idea is basically, [4:53] How text is just a sequence of letters. [4:56] And LLMs [4:57] learned really well how to predict the next, [4:59] and then they can complete the text. [5:01] Human actions are very similar, [5:03] but in our case, more importantly, companies [5:06] actions are similar. [5:07] You can represent a company as a sequence of actions. [5:10] The company is, I don't know, open credit line [5:13] ship product. [5:13] Click add and make changes. [5:15] They start accepting credit cards, take a loan. [5:18] And I mean these are large changes. [5:20] But all the myriad things that the company, [5:24] all of its employees, every second are doing, [5:27] it can only be turned into a sequence of actions. [5:30] And then what happens [5:32] is that you can create a digital twin of the company, [5:35] and then you can start doing things to it [5:37] instead of the real company and then see what happens. [5:41] You can have, counterfactual interventions. [5:43] You could say, oh, what if we added a advertising campaign? [5:47] Or what if they started shipping, [5:49] one day faster, or what if, [5:52] it's all about our merchants, what if we give them a loan [5:55] and then we find [5:56] a sequence of interactions that would maximize [5:59] their chances of growth? [6:01] And then we actually do that in production. [6:03] And so we go offer them a loan or we, [6:07] push them, send them communication. [6:11] Look for folks. [6:13] Your most important thing is [6:14] to, to start shipping faster or with, start advertising [6:18] campaign on Twitter or, So, having ability [6:22] to play with a company without endangering [6:26] anything, real people is is huge unlock. [6:30] Because now now you can [6:32] you can help more people to to be successful. [6:35] This is very clear. [6:37] Log. That wouldn't have happened. [6:39] And that has very direct impact on our bottom line. [6:43] And then the financial result. [6:45] And just building [6:46] this, let's say you could have built this, [6:47] how long would this have taken in the past [6:50] But this is the thing I don't think [6:51] it was possible in the past. [6:53] Right. [6:54] Just think about it [6:55] like you would need to get the concentration of talent. [6:58] You would have to bring that. [7:00] And then [7:00] and then you have [7:01] to have have this premonition that you need to do that, [7:05] and then you need to start [7:07] getting collecting data and all this, I don't think, [7:10] I don't think that was humanly feasible previously. [7:13] and so the trick here is the answer is like previously, [7:17] did they continue to we just wouldn't have done it. [7:20] And that's the [7:20] if my definition of raising the ceiling. [7:22] I think you start doing something that previously was [7:25] not even in consideration set. [7:27] What's something that you changed your mind about [7:30] because of because of Claude. [7:32] Because of the model. [7:33] I thought that the lens cannot really help me [7:38] manage the team better. [7:41] I could not have been more wrong. [7:43] And after a while, I realized, I, [7:47] I flipped so much, I created a system. [7:51] That actually analyzes what everything is happening like. [7:54] and proactively tells me, [7:57] But this it looks like this project is going to start [7:59] slipping there. Deadline. [8:00] it allows me to detect issues before they actually happen [8:04] and go over it [8:05] and then detect people unhappiness [8:06] and so the whole system starts functioning much better. [8:08] Has the role of it. [8:10] Yeah. But, Shopify changed. [8:12] now technical side is even more important [8:15] because the simple things I learned will go correct. [8:18] But, [8:19] the LLM is very good [8:20] at picking up signals and it tries to please. [8:23] And so whatever you ask, [8:25] the answer is very often in the question. [8:27] And it will just elucidate that. [8:29] And they give you exactly what you want to trade. [8:31] It's really important to want what's actually needed [8:34] not put in words first thing that comes to mind. [8:37] Because then then your consequences could be [8:40] actually worse than before, because before, at least [8:43] it wouldn't be immediately built. [8:44] There would be some discussion. [8:46] And now it's so much easier to just go and build it. [8:48] Technical acumen, becomes more important. [8:51] And the human component [8:52] of, supervising now, engineers who are supervising LLMs [8:56] rather than supervising engineers who are typing, [8:59] pressing buttons, [9:01] that's a very different thing. [9:02] That's it's almost like you get second, become second [9:06] level manager, because now you want to share. [9:09] Hey, what the best, best way is to dispatch the work [9:13] to different alarms? [unclear — possible "agents"?] [9:14] And how are you [9:14] going to not forget what they're doing [9:16] and then how are you going to avoid conflicts between them? [9:18] I think, [9:19] that's why I wouldn't say it's dramatically different [9:22] in, in that sense, but I say the accent is became [9:27] much more on those two things, like keeping [9:30] the team more efficient and making them more [9:34] have making technical decisions, [9:36] and less on checking day to day. [9:39] Hey, because that thing been done, [9:41] Okay, I want to end on this. [9:42] What is your advice to other CTOs, other [9:46] technical leaders that are also trying [9:49] to figure out how do I adopt quad? [9:51] How do I use it? [9:52] How do I scale it out to my org? [9:53] How do I get my work proficient? [9:55] my main advice would be try to [10:00] not limit tokens you always need, [10:02] like circuit breakers, because they're all these [10:04] runaway processes that suddenly start [10:06] burning tokens and and, doing things bad. [10:09] But it might not look like there's big difference between [10:13] big models and small models. [10:14] There's a world of difference and you want it. [10:18] You don't know the counterfactual, [10:19] so you want to know which one, which one is good. [10:22] Always use this dichotomy [10:25] barbell strategy for anything coding for anything. [10:29] Development, testing, auto research use. [10:33] The largest model budget for it is that cost for everything [10:38] production use, fine [10:40] tuning is the right model for that production. [10:42] Auto research the hell out of it [10:44] so that you balance both, cost, throughput and quality. [10:49] And save money on production by using cheaper models, [10:54] often fine tuned, [10:56] and invest that money in development site on the [10:59] on the place where there's huge potential in [11:03] upgrading your model to, in Fable 5, [11:06] I was on the record to say that it's [11:08] it was such a huge step, like much, much bigger than, [11:11] like all the previous steps. [11:13] Yes, I've seen repeatedly people doing the opposite [11:18] and trying to save money on, coding tokens on [11:21] unquote on development and then at the same time [11:26] wastefully running, more of those larger than they need [11:29] in the actual production services. [11:31] I feel like I will outcompete [11:32] anybody who is not on that stretch. [11:34] I was chatting with, my friends here that, Boris [11:38] destroyed my, my will to live freely in my, [11:41] I kept, loudly proclaiming that I will end up, [11:44] deciphering, Linear A, which is a very famous, [11:49] writing was used on Crete by Minoans, in the Bronze Age, [11:53] kind of personal area of interest of mine. [11:55] And then, and then Boris tweeted [11:56] that somebody did it using Claude, and I'm like, [11:58] damn it, that was my whole spiel. [12:00] I took my retirement project. [12:03] I was going to work on. [12:04] You had like, your old Commodore [12:04] I just didn't finish a time. [12:06] Exactly, exactly. [12:07] Thanks so much for stopping by.