The capability curve

The capability curve — Claude

Claude

AI-assisted summary of The capability curve — original source. Check the original for full context.

Source published:

Key ideas

  • Claude's score on SWE-bench Verified jumped from 62% on Claude 3.7 Sonnet to 87% on Claude Opus 4.7 over roughly twelve months, making it more than three times as likely to resolve difficult pull requests. (0:00)
  • Modern frontier models demonstrate three core qualitative improvements: deliberate upfront planning, self-healing error recovery that prevents repetitive doom loops, and coherent attention sustained across hundreds of thousands of tokens. (5:58)
  • Advancing model intelligence allows developers to prune legacy multi-step orchestration scaffolding and accumulated prompt instructions, often yielding better performance from a single execution thread with a cleaner prompt. (11:15)
  • Peak agent performance relies on closing the feedback loop—providing models with execution affordances like computer use so they can autonomously test, inspect, and iterate on their own outputs. (12:38)

Detailed notes

## Background Published on YouTube by the Claude channel on May 8, 2026, "The capability curve" is a 15-minute conference address presented by Anthropic research product manager Alex Albert. According to the video description, the session explores how rapidly frontier models are advancing, where this capability trajectory is headed, and what those shifts practically mean for developers building on Claude.

## TL;DR Alex Albert breaks down the rapid acceleration of AI coding over twelve months, spotlighting how Claude's SWE-bench Verified scores leaped from 62% on Claude 3.7 Sonnet to 87% on Claude Opus 4.7 and contrasting Sonnet 4's brittle output with Opus 4.7's functional single-prompt recreation of Claude.ai. He identifies three structural model leaps driving these gains: deliberative upfront planning, resilient error recovery that eliminates iterative doom loops, and sustained attention across long context windows. To harness this curve, Albert urges developers to prioritize realistic, unsaturated evaluations over synthetic benchmarks, pointing out that swapping in newer frontier models often outperforms complex manual optimization. He concludes by advising teams to aggressively prune accumulated system prompt debt and brittle scaffolding, utilize adaptive thinking and autonomous tool classifiers, and close feedback loops so agents can self-verify their implementations.

## Atmosphere & style The presentation carries an energetic, pragmatic, and engaging keynote vibe, opening with self-deprecating banter about competing with the venue's backstage happy hour. Albert commands the room with unhurried confidence, easily laughing off an on-stage video playback glitch and using live audience polling to anchor his points in real developer experiences. Rather than leaning on abstract hype, the talk feels like a grounded, tactical masterclass that invites engineers to trade brittle, over-engineered prompt scaffolding for simpler architectures that trust the model's expanding reasoning capabilities.

## Bulletin - [00:00](https://www.youtube.com/watch?v=tP4MGcJ80Y0&t=0s) Claude's score on SWE-bench Verified jumped from 62% on Claude 3.7 Sonnet to 87% on Claude Opus 4.7 over roughly twelve months, making it more than three times as likely to resolve difficult pull requests. - [05:58](https://www.youtube.com/watch?v=tP4MGcJ80Y0&t=358s) Modern frontier models demonstrate three core qualitative improvements: deliberate upfront planning, self-healing error recovery that prevents repetitive doom loops, and coherent attention sustained across hundreds of thousands of tokens. - [11:15](https://www.youtube.com/watch?v=tP4MGcJ80Y0&t=675s) Advancing model intelligence allows developers to prune legacy multi-step orchestration scaffolding and accumulated prompt instructions, often yielding better performance from a single execution thread with a cleaner prompt. - [12:38](https://www.youtube.com/watch?v=tP4MGcJ80Y0&t=758s) Peak agent performance relies on closing the feedback loop—providing models with execution affordances like computer use so they can autonomously test, inspect, and iterate on their own outputs.

## Key takeaways - [00:00](https://www.youtube.com/watch?v=tP4MGcJ80Y0&t=0s) Claude Code transitioned within roughly a year from an early, pre-GA prototype into a trusted, core development workflow delivering substantial speedups for developers. - [00:00](https://www.youtube.com/watch?v=tP4MGcJ80Y0&t=0s) SWE-bench Verified performance surged from 62% on Claude 3.7 Sonnet to 87% on Claude Opus 4.7, representing a 25+ percentage point leap in autonomous pull request resolution. - [03:41](https://www.youtube.com/watch?v=tP4MGcJ80Y0&t=221s) In a single-prompt recreation of Claude.ai, Sonnet 4 produced a crashing static mock-up, whereas Opus 4.7 produced a functional, branded app with live API integration and state management using fewer lines of code. - [05:58](https://www.youtube.com/watch?v=tP4MGcJ80Y0&t=358s) Unlike older models that impulsively executed code immediately, modern frontier models deliberate and structure implementation strategies upfront before executing. - [05:58](https://www.youtube.com/watch?v=tP4MGcJ80Y0&t=358s) Advanced error recovery enables agents to recognize dead ends and backtrack, eliminating context-wasting 'doom loops' of repetitive hallucinated patches. - [05:58](https://www.youtube.com/watch?v=tP4MGcJ80Y0&t=358s) Attention spans now remain coherent across hundreds of thousands to a million tokens, removing the need for developers to micromanage context windows or split tasks into fragile micro-steps. - [07:15](https://www.youtube.com/watch?v=tP4MGcJ80Y0&t=435s) Enterprise deployments validate these leaps: Vercel observed Opus 4.7 writing formal systems code proofs upfront, Windsurf recorded unbroken multi-step context fidelity, and Shopify saw iterative self-refinement. - [07:15](https://www.youtube.com/watch?v=tP4MGcJ80Y0&t=435s) Building reliable, production-aligned evaluation harnesses is the essential prerequisite for capturing the gains of frontier capability improvements. - [10:00](https://www.youtube.com/watch?v=tP4MGcJ80Y0&t=600s) Evaluations must reflect real user traffic distributions rather than academic benchmarks, and must be continuously hardened to remain unsaturated as models advance. - [10:00](https://www.youtube.com/watch?v=tP4MGcJ80Y0&t=600s) Swapping in the latest frontier model often outperforms weeks of complex manual application and prompt optimizations. - [11:15](https://www.youtube.com/watch?v=tP4MGcJ80Y0&t=675s) As reasoning improves, complex multi-step orchestration scaffolding can often be collapsed into a single execution thread to reduce fragility and latency. - [11:15](https://www.youtube.com/watch?v=tP4MGcJ80Y0&t=675s) System prompts tend to accumulate legacy technical debt over time and should be aggressively audited and pruned with every new model generation. - [12:38](https://www.youtube.com/watch?v=tP4MGcJ80Y0&t=758s) Adaptive thinking allows models to decide autonomously when and how deeply to deliberate, regulated by developers via the effort parameter. - [12:38](https://www.youtube.com/watch?v=tP4MGcJ80Y0&t=758s) Claude Code's Auto Mode utilizes classifiers to safely distinguish benign tool calls from actions requiring explicit human approval, enabling background autonomy. - [12:38](https://www.youtube.com/watch?v=tP4MGcJ80Y0&t=758s) Developers should grant agents affordances for self-inspection—such as computer use for UI verification—to let them autonomously critique and correct their work.

## Detailed walkthrough ### [00:00](https://www.youtube.com/watch?v=tP4MGcJ80Y0&t=0s) The Rapid Rise of Agentic Coding and SWE-bench Gains

Anthropic research product manager Alex Albert takes the stage, first setting his water bottle down and joking that he knows a conference happy hour just kicked off backstage. As Alex Albert quips to the audience, “I just heard from the stage manager alone that there's a happy hour that just started. So I want to thank all you guys for choosing me over drinks and snacks.” Reflecting on the shift in atmosphere over the past twelve months, he notes that during the previous year's event, Claude Code had only been out for two or three months and had not even reached General Availability (GA). Developers were only beginning to wrap their heads around the paradigm of agentic coding. A year later, developers are placing real trust in Claude, shipping software significantly faster, and tackling projects that previously felt impossible to complete within practical timeframes.

To take the pulse of the room, Albert asks attendees to participate in an informal hands-up exercise. A large portion of the audience keeps their hands raised when asked if Claude has made them go ten times faster compared to a year ago, followed by widespread affirmation for 5x, 2x, and general day-to-day usage of Claude. Moving from qualitative impressions to rigorous evaluation, Albert points to SWE-bench Verified, a benchmark designed to evaluate a model's capacity to autonomously resolve and merge software pull requests. Roughly a year prior, Claude 3.7 Sonnet scored 62% on the benchmark. With Claude Opus 4.7, that figure surged to 87%—a jump of more than 25 percentage points in just over twelve months. As Alex Albert explains the practical significance of that metric, “Opus 4.7 is more than three times as likely to succeed on some of those difficult PRs that Sonnet 3.7 was failing on a year ago.”

- **Claude Code evolution**: Transitioned from an early pre-GA prototype out for only 2–3 months into a trusted, core development workflow in roughly a year. - **SWE-bench Verified progress**: Jumped from 62% on Claude 3.7 Sonnet to 87% on Claude Opus 4.7. - **Real-world multiplier**: Represents a more than 3x improvement in success rate on complex software engineering pull requests that older frontier models routinely failed.

---

[03:41](https://www.youtube.com/watch?v=tP4MGcJ80Y0&t=221s) Demo: Recreating Claude.ai in a Single Prompt

To show what a 25% benchmark jump feels like in practice rather than in abstract percentages, Albert presents a side-by-side comparison of two models tested on the exact same task twelve months apart: Claude Sonnet 4 versus Claude Opus 4.7. After a brief on-stage technical hiccup where the video playback stalls—prompting Albert to joke that the audience might just have to take his word for how cool it was—the demo fires up inside Claude Code. The challenge given to each model is ambitious: recreate the full Claude.ai web interface from scratch using only a single prompt.

Sonnet 4 approaches the task with surface-level competence but shallow execution. It produces a generic, stark black-and-white chat user interface. However, the moment a user submits a prompt, the application immediately throws an unhandled error and crashes; it functioned merely as a static mock-up rather than an operable product. By contrast, Opus 4.7 leverages Claude Code to inspect requirements, invoke a flurry of tools, and write functional code end-to-end. Its output accurately mirrors the official Claude branding and color scheme, successfully establishes an active connection to the Claude API, receives and streams real model responses, manages conversation history so prior threads remain intact when starting new chats, renders inline visualizations just like the production Claude.ai platform, and implements dark mode. Furthermore, Opus 4.7 achieves all of this functionality using fewer total lines of code than its predecessor.

```mermaid flowchart TD accTitle: Single-Prompt Claude.ai Recreation Comparison accDescr: Flowchart contrasting the execution paths and results of Sonnet 4 versus Opus 4.7 when prompted to recreate Claude.ai.

subgraph Prompt["Single Input Prompt"] P["'Recreate Claude.ai'"] end

subgraph Sonnet["Sonnet 4 (12 Months Ago)"] S1["Writes basic markup"] --> S2["Generic black-and-white UI"] S2 --> S3["Submits message"] S3 --> S4["Immediate crash / unhandled error"] end

subgraph Opus["Opus 4.7 (Current Frontier)"] O1["Executes multiple tool calls"] --> O2["Builds Claude-branded UI & Dark Mode"] O2 --> O3["Integrates live Claude API"] O3 --> O4["Preserves chat history & renders inline visualizations"] end

P --> Sonnet P --> Opus ```

- **Task specification**: Reconstruct a fully functional clone of the Claude.ai application from a single prompt within Claude Code. - **Sonnet 4 outcome**: Fragile UI shell with no functional backend connectivity that fails instantly upon user input. - **Opus 4.7 outcome**: Production-like replica featuring brand styling, active API communication, persistent multi-chat state management, native visualization rendering, dark mode, and higher code efficiency.

---

[05:58](https://www.youtube.com/watch?v=tP4MGcJ80Y0&t=358s) Three Core Leaps: Planning, Error Recovery, and Long-Context Attention

Shifting focus away from benchmark scoreboards or individual model releases, Albert emphasizes that developers must understand the underlying structural shifts taking place across the frontier capability curve. As Alex Albert frames the central theme of his address, “What I do want to focus on is what it means to build on something that is getting meaningfully better like this month over month.” Across the latest model generations, these qualitative leaps are concentrated in three specific areas: upfront planning, error recovery, and sustained attention across extended runs.

The first major breakthrough is in deliberative planning. Older generations suffered from an impulsive failure mode where they would immediately begin executing code before understanding the scope of the problem. Albert compares this behavior to his own flawed habit of assembling flat-pack IKEA furniture without reading the manual, only stopping to inspect the instructions once the living room is already an unmanageable mess. Newer frontier models invert this tendency by taking dedicated time upfront to reason through edge cases, formulate an architecture, and only then write implementation code. Because forcing an agent to jump straight into tool use degrades downstream output quality, Albert urges developers to allocate explicit inference budget and time for Claude to formulate its plan.

The second area of advancement is error recovery and the elimination of the dreaded "doom loop." When older agents encountered unexpected errors or failing tests, they frequently proposed flawed fixes, failed again, and spiraled into repetitive, hallucinated patches until the context window stalled out entirely. Newer models exhibit adaptability: they recognize when an approach is failing, step back to re-evaluate their assumptions, and explore alternate architectural paths. This resilience directly translates to higher task completion rates and vastly fewer wasted tokens. Finally, models have achieved dramatic improvements in sustaining attention over long horizons. Whereas older models would "lose the plot," forget system prompt constraints, or drift as token counts mounted, modern models maintain strict coherence across hundreds of thousands or even up to a million tokens. Developers no longer need to micromanage context windows or artificially fragment workflows into fragile micro-tasks.

- **Upfront planning**: Models deliberate and structure implementation strategies prior to execution; as Alex Albert advises developers, “What this means for you as a developer is that you should allow Claude that time to think.” - **Resilient error recovery**: Modern agents actively backtrack out of dead ends rather than spiraling into context-exhausting doom loops, preserving token budgets. - **Extended attention spans**: Strong adherence to system instructions and problem context over hundreds of thousands to one million tokens eliminates the need to constantly babysit context windows.

---

[07:15](https://www.youtube.com/watch?v=tP4MGcJ80Y0&t=435s) Customer Evidence and the Shift to Evals

These three compounding capabilities—deliberate planning, self-healing error recovery, and persistent long-run attention—are already translating into tangible wins across enterprise software teams. Albert highlights several real-world developer platform examples: - **Vercel**: Observed Opus 4.7 independently drafting formal mathematical and architectural proofs for low-level systems code before writing a single line of actual implementation. - **Windsurf**: Discovered in their benchmark evaluations that Claude held unbroken attention and context fidelity over their longest, most demanding multi-step agentic runs. - **Shopify**: Found that rather than blindly generating code and moving on, Claude actively looped back to review, test, and iteratively refine its own outputs.

To help developers replicate these enterprise-grade breakthroughs in their own products, Albert argues that engineering teams must shift their primary focus. The journey toward harnessing frontier capabilities does not begin with user-facing application logic or elaborate prompt engineering, but with evaluation harnesses. As Alex Albert succinctly emphasizes, “If you can measure something, you can improve on it.” Establishing reliable, automated evals forms the bedrock requirement for steering and capturing the benefits of the rapidly accelerating model capability curve.

[10:00](https://www.youtube.com/watch?v=tP4MGcJ80Y0&t=600s) Building Unsaturated Evals and Swapping Frontier Models

A foundational requirement for developers building on frontier models is ensuring that their evaluations actually reflect reality. Developers frequently stray by evaluating their application on synthetic or academic benchmarks that are merely adjacent to their actual use case rather than on their true task distribution. For instance, teams building a coding agent often benchmark it against standard academic benchmarks rather than testing it against real user traffic or realistic traffic patterns. In addition to measuring the right distribution, evaluations must be kept unsaturated. As models become more intelligent, benchmarks quickly saturate and lose their utility unless they are continuously made harder to match the advancing capability frontier.

Once rigorous, unsaturated evaluations are in place, the most effective step is often straightforward: test them immediately against the newest frontier releases. Developers frequently spend weeks engineering complex prompt or workflow tweaks when an upgraded model could solve the problem out of the box. As the speaker highlights: “I found that sometimes the best optimization you can make to your app is simply swapping in the latest model.”

- Measure the actual production distribution rather than relying on detached academic benchmarks. - Continually update and harden evaluations so they scale alongside frontier intelligence and provide meaningful signal. - Regularly benchmark new model releases, as a raw model upgrade can outshine complex manual application optimizations.

---

[11:15](https://www.youtube.com/watch?v=tP4MGcJ80Y0&t=675s) Pruning Scaffolding and Cleaning Up Prompts

Developers need to fundamentally reconsider their application scaffolding—the code, prompts, skills, and tool setups that surround the model to guide it toward its objectives. As models grow more capable, historical architectural workarounds become obsolete. Instead of maintaining an elaborate, brittle multi-step workflow across different agents or sub-tasks, teams can often collapse the task into a single execution thread and let the model handle the reasoning internally. Removing structural complexity frequently produces a net increase in overall performance.

System prompts suffer from an identical issue over time. Prompts accumulate rules, edge-case patches, and instructions across multiple model generations until they degrade into legacy technical debt. As the speaker explains: “After a long amount of time and many different model generations, it becomes somewhat of like a hideous mess of different rules and instructions.” With every frontier model upgrade, developers should audit their system prompts to strip away instructions that newer models adhere to naturally, which simultaneously sharpens task execution and reduces token overhead.

```mermaid flowchart TD accTitle: Scaffolding and Prompt Simplification accDescr: Moving from legacy multi-step scaffolding to streamlined single-thread execution with pruned prompts.

subgraph Legacy["Legacy Architecture"] A["Accumulated System Prompt<br/>('Hideous mess' of rules)"] --> B["Multi-Step Workflow Scaffold"] B --> C["Brittle Tool & Orchestration Layers"] end

subgraph Modern["Frontier Model Architecture"] D["Pruned, Minimalist Prompt"] --> E["Single Execution Thread"] E --> F["Autonomous Tool Use & Reasoning"] end

Legacy -.->|"Upgrade to newer frontier model"| Modern ```

---

[12:38](https://www.youtube.com/watch?v=tP4MGcJ80Y0&t=758s) Giving the Model Room: Adaptive Thinking, Auto Mode, and Closing the Loop

To extract peak performance from modern models, developers must give them sufficient room to work. This begins by leveraging adaptive thinking and letting Claude decide when and how deeply to plan, using the effort parameter to regulate the balance between thinking tokens and operational actions. Beyond reasoning autonomy, models should be granted wider access to external tools in a controlled, safe manner. Rather than requiring human confirmation for every action or giving the agent unchecked permissions, systems can implement intelligent classification barriers. In Claude Code, for example, Auto Mode applies classifiers over proposed tool calls to determine autonomously whether a command requires explicit human authorization, allowing the agent to sustain background execution safely without constant interruptions.

Finally, developers must close the feedback loop so that agents can inspect their own work and iterate. Because models have become vastly better at critique and verification, systems should provide the affordances necessary for self-correction. For a coding agent writing frontend code, this means equipping it with computer use capabilities so it can interact with the rendered user interface, click around, and QA test its own implementations. Wrapping up the session, the speaker encourages attendees to rethink their systems around these expanding boundaries: “Models are continuously getting better at verifying and iterating on its outputs, so it's important to allow it the affordances to do so.”

- **Adaptive Thinking**: Let the model choose when to think and adjust reasoning depth via the effort parameter. - **Claude Code Auto Mode**: Classifies proposed tool calls to determine if explicit human approval is necessary, enabling safe autonomous execution in the background. - **Closed-Loop Verification**: Provide concrete feedback tools—such as computer use for frontend testing—so the agent can validate and refine its own outputs.

## Notable quotes > [10:00](https://www.youtube.com/watch?v=tP4MGcJ80Y0&t=600s) I found that sometimes the best optimization you can make to your app is simply swapping in the latest model. — Alex Albert

> [12:38](https://www.youtube.com/watch?v=tP4MGcJ80Y0&t=758s) Models are continuously getting better at verifying and iterating on its outputs, so it's important to allow it the affordances to do so. — Alex Albert

## Mentioned Alex Albert, Anthropic, Claude, Claude 3.7 Sonnet, Claude Code, Claude Opus 4.7, Claude Sonnet 4, Claude.ai, IKEA, Shopify, SWE-bench Verified, Vercel, Windsurf

## Video description Frontier models are getting more capable, fast. Where the curve is going, and what it means for developers building on Claude.

[Source](https://www.youtube.com/watch?v=tP4MGcJ80Y0)

Reading summary

Anthropic research product manager Alex Albert argues that rapid structural improvements in frontier models—specifically in planning, error recovery, and context fidelity—require developers to dismantle complex legacy scaffolding and shift engineering efforts toward unsaturated evaluations and autonomous, closed-loop execution. Compounding Gains in Agentic Coding Performance (0:00) Over roughly twelve months, agentic coding progressed from early experimental prototypes into core development workflows. Albert points to SWE-bench Verified benchmark scores, where model performance jumped from 62% on Claude 3.7 Sonnet to 87% on Claude Opus 4.7. According to Albert, this increase of more than 25 percentage points indicates that the newer model is more than three times as likely to resolve difficult pull requests that previously failed. To illustrate the practical difference between generations, Albert demonstrated recreating the Claude.ai interface from a single prompt within Claude Code. While Sonnet 4 generated a fragile black-and-white shell that crashed on the first user submission, Opus 4.7 produced an operational application using fewer total lines of code. The newer output successfully established live API connections, streamed responses, maintained persistent chat history across sessions, rendered inline visualizations, and incorporated official styling and dark mode. Structural Leaps in Reasoning, Recovery, and Context (5:58) Albert states that the capability curve is driven by qualitative shifts across three distinct mechanisms rather than minor incremental adjustments. First, models now engage in upfront deliberative planning, formulating architectures and mapping edge cases before executing code, in contrast to earlier generations that jumped prematurely into tool calls. Second, models demonstrate resilient error recovery, stepping back to reassess flawed assumptions rather than entering repetitive, context-wasting "doom loops" when tests or commands fail. Third, long-context attention spans have expanded to sustain instruction adherence across hundreds of thousands or up to a million tokens without losing coherence. Enterprise platform adoption reflects these traits: Albert noted that Vercel observed Opus 4.7 drafting formal mathematical and architectural proofs prior to writing low-level systems code, Windsurf recorded continuous fidelity over extended multi-step runs, and Shopify saw models actively reviewing and iteratively refining their own code. Prioritizing Unsaturated Evaluations and Model Swaps (10:00) Capitalizing on frontier improvements requires engineering teams to ground development in rigorous evaluation harnesses rather than manual prompt tweaks. Albert warns that developers frequently evaluate applications against synthetic academic benchmarks that diverge from real-world traffic distributions. Because advancing models quickly saturate standard benchmarks, evaluations must be continuously hardened and updated to reflect actual production complexity. Maintaining unsaturated evals allows teams to test new model releases immediately. Albert observes that developers often spend weeks building brittle application workarounds when simply swapping in the latest frontier release would solve the underlying problem out of the box with zero custom logic. Pruning Scaffolding and Closing Feedback Loops (11:15) As models internalize reasoning and planning, surrounding application scaffolding should be simplified. Multi-agent chains and rigid multi-step orchestration workflows can frequently be collapsed into single execution threads. Similarly, system prompts that have accumulated rules, edge-case patches, and instructions over multiple model generations turn into technical debt and should be pruned to lower token overhead and sharpen execution. To maximize performance, Albert advises giving models room to operate by enabling adaptive thinking parameters and granting wider tool access. In Claude Code, an "Auto Mode" uses classifiers over tool calls to safely determine whether actions require explicit human sign-off or can run autonomously in the background. Finally, Albert stresses closing the feedback loop with testing affordances—such as computer use capabilities for frontend verification—so models can inspect, QA test, and iteratively correct their own work. Albert concludes that as frontier models advance in self-verification and critique, engineering value moves away from writing rigid constraints and toward designing realistic evaluation environments that grant models the autonomy and tool affordances needed to self-correct.

Source quotes

[10:00] Alex Albert: I found that sometimes the best optimization you can make to your app is simply swapping in the latest model. [12:38] Alex Albert: Models are continuously getting better at verifying and iterating on its outputs, so it's important to allow it the affordances to do so.

Original transcript

[0:00] Thank you. Thank you. Bye. . . Thank you. Thank you. you Bye. *music* Bye. - Please welcome to the stage, member of technical staff of Anthropic, Alex Albert. - Hello, hello everybody. All right, let me set my water down here. I'm Alex Albert. I'm one of our research PMs here at Anthropic. I just heard from the stage manager alone that there's a happy hour that just started. So I want to thank all you guys for choosing me over drinks and snacks. I really appreciate that. And I promise this will be worth your 15-20 minutes here. Um, It's been really interesting comparing the vibes at this year's conference to last year's conference. Last year, Cloud Code was not even GA'd. It had been out for about two or three months. Folks were just starting to get used to this paradigm of agentic coding. These days, as I'm talking to folks here, it feels like people are doing a lot more They're trusting Claude. they're shipping things faster with Claude. They're building things that they weren't building before that haven't been possible to build on the time scales that they are. Actually out of curiosity, I want to run a little exercise here. So I'm gonna say a question. If the question is you, please raise your hand and just keep your hand up. I want to get a sense of the room. Raise your hand if you feel like Claude has allowed you to go 10 times faster. than what you were doing a year ago. Okay, wow, that was a lot. Keep it up. How about five times faster? Thank you. All right, 2x faster. Raise your hand if you use Claude. There we go, okay, love that. One of the best ways that we've measured Claude's coding intelligence and how we've seen Claude make this impact on folks is through this benchmark called SWE Bench Verified. Sweetbench Verified measures the model's ability to autonomously complete software PRs. About a year ago on our model at the time, Sonnet 3.7, it scored 62% on this eval. Today, with Opus 4.7, it scores 87%. Yeah. That's a over 25% jump in just over a year. To put this in other words... Opus 4.7 is more than three times as likely to succeed on some of those difficult PRs that Sonnet 3.7 was failing on a year ago. Now, numbers are great, but examples are even better. So to make this a bit more concrete, I have a quick demo here that we put together of the same task, but 12 months apart. So let's get into it. Thank you. So in this example, we're going to be comparing Sonnet 4 to Opus 4.7. and we're gonna give them the same task. Oh. Let's go back here and get this demo working. I'll give this a second here to see if it's playing. Thank you. If the people watching this... Oh, there we go. Wait. No. Still nothing. All right. Well... [5:00] There was a demo here and it was really, really cool, so you're gonna have to believe me on that. Oh, here we go. Okay. Sonnet 4, working within Cloud Code. The task that we gave it was to recreate Claw.ai with a single prompt. So you can see how it does here. We get this really generic black and white chat application. We can enter in a prompt. We'll fire that off and immediately we just hit an error. So it doesn't really even work. Basically, it just made a nice little UI for us. Now let's run this same task but with Opus 4.7 instead and see how it does. So we're going to have the same setup. It's going to be running within Cloud Code. It's going to be writing a bunch of lines of code, calling a ton of tools, Thank you. And eventually we're going to get an output. And immediately this output looks better. So we have the Claude color scheme. It knows that already. We actually get a response back from the Cloud API when we send it a message. We can start a new chat and it still remembers the old chat. So we're keeping track of things. It even renders in-line the visualization, like the Claw.ai app does, And like a true developer, it even implemented dark mode. So in addition to that being a better output, it also did it in less lines of code, so it's more efficient as well. Thank you. Now, this talk isn't about any one of these models in particular. What I do want to focus on is what it means to build on something that is getting meaningfully better like this month over month. Now before I get into the specific tips, I want to take a look at where these model gains are actually landing. Starting first with Claude's planning ability. Thank you. Older models would have this particularly bad failure mode where they would act first and then think later. Thank you. It's kind of like me when I'm having to build like a new set of IKEA furniture, like I'm going straight into it. And then only once it's like a mess am I coming back and be like, "Ah, maybe I should actually read the instructions." I don't know if anybody else is the same on that, but that was basically the same as Sonnet 3.7. Newer models are... much more thorough. They take their time up front to actually think about the problem, strategize a little bit, plan out what they're going to do, and then dive into the action and dive into writing those lines of code. What this means for you as a developer is that you should allow Claude that time to think. Give it some time to actually think about the problem, what it's going to do. Don't force it to just jump straight into that action because this can reduce your downstream performance. Thank you. The second major area where we're seeing gains is in Claude's air recovery. Previous models would hit this thing that we'd often call a doom loop. So the model would run into a problem. It would try to propose a solution. That solution wouldn't work. And then all of a sudden it's really stuck. And it's just going to keep spiraling and spiraling until eventually that context stalls out and you got to just clear everything. Newer models are much more adaptable. If they hit a problem, they're able to now backtrack out of it and actually think about it in a different way and maybe take a different path. Yeah. What this means for you as a developer is that you're going to get better task performance from Claude with fewer wasted tokens because it's not going to get caught in those loops. The final area we're seeing these gains is in Claude's attention over long runs. Older models would often lose the plot as they're working on something. Maybe they would start forgetting things. They're not paying attention as much. Those instructions that you put in the system prompt aren't being paid attention to as much on that thread. Newer models are much more able to hold coherence across runs. They're able to remember those system prompt instructions and stay focused over the course of hundreds of thousands or even a million tokens. In terms of your applications and how you use Claude, this means that you don't need to babysit the context window as much. You don't need to chunk up work and you can trust that Claude can operate for you on long runs autonomously. Thank you. Adding these things all up together, we have better planning, we have fewer failures in error recovery, and we have agents running for longer. And this compounds into better end-to-end task performance. Our customers are seeing this as well. Vercel saw in Opus 4.7's planning that it was actually writing proofs for systems code before implementing a single line of code. Winsurf saw in their evals that Claude was really sustaining its attention over their longest agentic runs. And Shopify found that as the model was coding, it was actually going back and iteratively refining its outputs. So how can you as a developer see those same things our customers are seeing in your applications? Well, somewhat counterintuitively, it starts not with your application, but with your evals. If you can measure something, you can improve on it. So it's important that, A, you have [10:00] else. And B, these evals are measuring something close to your product distribution that you actually want to improve on. So That might sound pretty obvious, but it's often something that I see leads developers astray. When they try to eval their product on something adjacent to their use case, but not on their actual task distribution. So maybe for example they have a coding agent and instead of evaling that coding agent on traffic they have or traffic of a similar pattern of what their users are doing, they're evaling it on an academic coding benchmark. The second thing you want to do once you have those evals is to make sure that they're not saturated. As models are getting smarter, evals are continuously needing to get harder and harder in order to get more signal from Frontier models. You want to make sure that the evals you're building are growing alongside the model. so that you can continue to get new signal as more intelligent models come out. And finally, once you have those evals and you've been sure they're not saturated, test them on the newest Frontier models. I found that sometimes the best optimization you can make to your app is simply swapping in the latest model. So it's worth spending a meaningful amount of time testing and trying out models as they come. Thank you. The second tip for you here is to take a second look at your scaffolding. When I'm saying scaffolding, what I'm referring to is the code and the prompts and the skills and the tool setups, everything that kind of surrounds the model and directs it towards its goals. Thank you. With newer models, you might not need some of the things that you needed before. So maybe instead of a multi-step workflow, you can just let the model work on a task in one thread. Thank you. Often you can actually boost your performance by removing instead of adding things onto your scaffolding. Now, alongside your scaffolding, you also want to take a second look at your prompts. Prompts begin to build up model over model. After a long amount of time and many different model generations, it becomes somewhat of like a hideous mess of different rules and instructions. And you're not really sure why you added one thing and what it's doing now for your future models. Thank you. with every new model, take a second look at those prompts, cut down on things that might not be needed anymore. And this will help both your task performance but also save you tokens as well. Final tip here is to give the model room to work. Thank you. Like I was mentioning earlier when I was talking about planning, it's important to let Claude choose when to think. Use adaptive thinking. and dial the amount of tokens that Claude is thinking, and the amount of actions that it's taking with the effort parameter. Second thing here is to allow your agents more access to tools in a controlled way. Now, some of you hearing that might get a little bit nervous. I'm not saying to just let it do anything. But there are ways and methods that you can take to allow Claude to actually execute on more systems in a safe way. One example of this is Claude Code Auto Mode. So in one of our recent engineering blog posts, we talked about Auto Mode. Auto Mode actually runs classifiers over the tool calls that Claude is proposing, and we can determine if that tool call needs explicit human approval or not. This allows us to let Claude run in the background for longer, and it can work more autonomously without needing a human to step in. The final tip here is to ensure that you're closing the loop for your agent. Design your system so that Claude can actually inspect its own outputs and iterate on them. Going back to that coding agent example, if you have an agent that's working on frontend applications, maybe you want to give it a computer use tool so that it can click around on the site and Q&A test different bugs or features that it writes. Models are continuously getting better at verifying and iterating on its outputs, so it's important to allow it the affordances to do so. And with that, that is a quick look at the capability curve. I want to thank everybody for coming out today. I know this is the last talk of the day. I'll be hanging around in the reception, so please come find me. I'd love to chat about how we can make Claude better for you. Thank you. Bye. *music* Oh, oh, oh. [15:00] you Hey.