Browserbase Automates Agent Training to Bypass Web Obstacles

Hill-Climbing Skills: Improve Agents Without Changing the Model — Shubhankar Srivastava, Browserbase — AI Engineer

AI Engineer

Persistent strategy files enable agents to learn from execution traces, slashing operating costs and bypassing anti-bot defenses without retraining model weights.

AI-assisted summary of Hill-Climbing Skills: Improve Agents Without Changing the Model — Shubhankar Srivastava, Browserbase — original source. Check the original for full context.

Source published:

Key ideas

  • Browserbase introduced AutoBrowse, an architecture pairing a 'doer' agent with a 'teacher' agent that inspects run traces and updates an external strategy file, enabling agents to hill-climb and learn without retraining model weights. (6:17)
  • Persisting execution lessons across runs achieved 3x to 16x cost reductions across production tasks, such as dropping a Google Flights search from $12 to $3 and an OpenTable reservation from $6.22 to $0.51. (11:26)
  • Porting the AutoBrowse loop to non-browser domains such as coding or deep research requires two fundamental prerequisites: a fully replayable environment and a verifiable outcome. (20:28)
  • Graduated browsing skills are stored as portable Markdown files decoupled from vendor tooling, allowing direct execution inside alternative harnesses like Anthropic's Claude Code CLI. (45:19)

Detailed notes

## Background Published on YouTube by AI Engineer on October 11, 2026, this talk features Shubhankar Srivastava of Browserbase presenting in the final workshop session of the day. The presentation covers how browser agents can improve reliability without modifying model weights by using an AutoBrowse loop—pairing a browser-driving doer agent with a reflective teacher agent that extracts lessons into strategy files and graduates them into reusable skills.

## TL;DR Srivastava introduces an iterative hill-climbing framework inspired by Andrej Karpathy's auto-research loop to overcome the reliability bottlenecks of browser agents. By pairing a doer agent executing browser actions with a teacher agent reviewing traces and screenshots, the system captures selectors, URL shortcuts, and anti-bot workarounds into persistent strategy files rather than starting every task from scratch. Demonstrations across Google Flights, OpenTable, and conference agendas illustrate cost reductions between 3x and 16x and show how accumulated knowledge stabilizes into portable, reusable markdown skills. The session also addresses the prerequisites for extending the framework to other domains, managing website drift via automated repair loops, and deploying layered security controls against prompt injection.

## Atmosphere & style The workshop has the energetic, pragmatic vibe of an experienced practitioner demystifying the messy edge cases of autonomous browser automation. Srivastava delivers an engaging, conversational presentation characterized by candid commentary on agent blunders, transparent live demos where intermediate runs bounce between success and failure, and enthusiastic audience banter over Slido. It feels like an open-ended engineering post-mortem combined with a live hacking session, emphasizing the iterative grind required to turn fragile web agents into hardened production tools.

## Bulletin - [06:17](https://www.youtube.com/watch?v=p7gaxmCUiIU&t=377s) Browserbase introduced AutoBrowse, an architecture pairing a 'doer' agent with a 'teacher' agent that inspects run traces and updates an external strategy file, enabling agents to hill-climb and learn without retraining model weights. - [11:26](https://www.youtube.com/watch?v=p7gaxmCUiIU&t=686s) Persisting execution lessons across runs achieved 3x to 16x cost reductions across production tasks, such as dropping a Google Flights search from $12 to $3 and an OpenTable reservation from $6.22 to $0.51. - [20:28](https://www.youtube.com/watch?v=p7gaxmCUiIU&t=1228s) Porting the AutoBrowse loop to non-browser domains such as coding or deep research requires two fundamental prerequisites: a fully replayable environment and a verifiable outcome. - [45:19](https://www.youtube.com/watch?v=p7gaxmCUiIU&t=2719s) Graduated browsing skills are stored as portable Markdown files decoupled from vendor tooling, allowing direct execution inside alternative harnesses like Anthropic's Claude Code CLI.

## Key takeaways - [02:41](https://www.youtube.com/watch?v=p7gaxmCUiIU&t=161s) Production browser agent failures stem from an amnesia problem: agents start from scratch every run without episodic memory to learn from execution traces. - [06:17](https://www.youtube.com/watch?v=p7gaxmCUiIU&t=377s) The AutoBrowse architecture pairs a doer agent executing browser commands with a teacher agent that evaluates traces and writes lessons into an evolving strategy.md file. - [07:13](https://www.youtube.com/watch?v=p7gaxmCUiIU&t=433s) Embracing Rich Sutton's 'Bitter Lesson,' the system allows iterative compute and trial-and-error to discover unintuitive shortcuts, such as encoding search parameters directly into URLs. - [13:06](https://www.youtube.com/watch?v=p7gaxmCUiIU&t=786s) Autonomous trial-and-error can uncover environmental workarounds, such as appending specific query parameters to bypass OpenTable's anti-bot 'Access Denied' barriers. - [11:13](https://www.youtube.com/watch?v=p7gaxmCUiIU&t=673s) Once performance stabilizes, accumulated strategy files graduate into permanent, reusable skill files that eliminate exploratory costs in production runs. - [15:48](https://www.youtube.com/watch?v=p7gaxmCUiIU&t=948s) AutoBrowse workflows amortize generation costs—typically between $30 and $40 per skill—across thousands of production runs, yielding consistent 3x to 16x operational savings. - [20:28](https://www.youtube.com/watch?v=p7gaxmCUiIU&t=1228s) Extending hill-climbing architectures to other domains requires a replayable, grindable environment that can easily reset state, alongside a quantifiable verification metric. - [26:52](https://www.youtube.com/watch?v=p7gaxmCUiIU&t=1612s) Running a cheaper model like Claude 3.5 Sonnet over more iterations often matches or exceeds the quality of fewer Opus runs at a lower total generation cost. - [42:38](https://www.youtube.com/watch?v=p7gaxmCUiIU&t=2558s) Solving anti-bot barriers and CAPTCHAs natively at the custom Chromium binary layer avoids burdening LLM token budgets or reasoning cycles. - [39:26](https://www.youtube.com/watch?v=p7gaxmCUiIU&t=2366s) Teacher agents can be configured for diverse objectives, including optimizing pure reliability, minimizing turn counts, or discovering raw HTTP endpoints to bypass DOM interaction entirely. - [37:54](https://www.youtube.com/watch?v=p7gaxmCUiIU&t=2274s) Website drift is handled through hybrid maintenance: lightweight health checks monitor critical paths, while production failures trigger automated repair loops that open pull requests. - [45:19](https://www.youtube.com/watch?v=p7gaxmCUiIU&t=2719s) Graduated skills export as human-readable Markdown files detailing navigation rules and selectors, making them completely vendor-agnostic and runnable in any agent framework. - [47:30](https://www.youtube.com/watch?v=p7gaxmCUiIU&t=2850s) Defending browser agents against prompt injection requires layered controls: pre-harness query validation, automated skill security reviews, and infrastructure-level network sandboxing.

## Detailed walkthrough ### [00:00](https://www.youtube.com/watch?v=p7gaxmCUiIU&t=0s) Improving Browser Agents Without Changing Model Weights

Opening the final workshop of the day, Shubhankar Srivastava introduces the core thesis of his presentation: improving autonomous agents without modifying underlying model weights or retraining models. Shubhankar, who works on the sales engineering team at Browserbase, spends his day-to-day helping enterprise customers build and deploy browser agents in production environments. Before diving into technical mechanics, he gauges the room with a quick show of hands to see how many attendees are familiar with browser agents or have actively built them using available tooling.

To establish common ground, Shubhankar defines a browser agent simply as an autonomous agent that uses a web browser as an execution tool. Rather than interacting purely through text completions, the agent drives browser actions through an interface—whether a Command Line Interface (CLI), the Model Context Protocol (MCP), or programmatic scripts. Through these control hooks, the agent issues low-level commands such as clicking a specific button, triggering a page refresh, capturing screenshots, or taking snapshots of the site's accessibility tree. Browserbase acts as the underlying cloud infrastructure platform for these workflows, providing the remote browser environments, SDKs, CLIs, and execution skills necessary to run browser automation reliably at scale.

- **Speaker Role:** Shubhankar Srivastava, Sales Engineering at Browserbase. - **Audience Context:** Final workshop session of the conference day, featuring an interactive Slido QR code for live audience Q&A. - **Core Technology Stack:** Browser automation tools, MCP connectors, low-level browser CLIs, and Browserbase remote browsing infrastructure.

---

[02:41](https://www.youtube.com/watch?v=p7gaxmCUiIU&t=161s) The Reliability Crisis: Geniuses That Never Learn

Moving to the central pain point of modern web automation, Shubhankar highlights that reliability remains the single biggest hurdle preventing browser agents from succeeding in production. Developers routinely see agents get stuck in infinite interaction loops, trigger anti-bot or CAPTCHA challenges, drop context mid-task, or simply exhibit "laziness" by giving up before reaching the goal. While recent foundation models like Claude Opus demonstrate immense raw intelligence, Shubhankar emphasizes that making them dependable cannot be solved by model intelligence alone; it requires robust scaffolding, tooling, and execution skills around the model.

To illustrate how fragile agents are today, Shubhankar walks through two common failure modes. In one example, an agent navigating Google Flights attempts to adjust flight departure times using a slider element. Because dragging a physical slider requires precise coordinate targeting or specific JavaScript drag events, the agent becomes utterly confused, fails to isolate the underlying HTML elements, and ultimately abandons the booking. In another scenario, an agent attempting to reserve a date night dinner on OpenTable crashes headfirst into an anti-bot "Access Denied" barrier.

When diagnosing the root cause, Shubhankar argues that the fundamental flaw is that agents possess no episodic memory or persistence across attempts. Fine-tuning models on specific sites like OpenTable is inherently unscalable because websites change constantly and new tasks emerge continuously. Instead, agents must be given a mechanism to learn directly from their production execution traces.

> "Essentially, there's no learning happening. Every agent run is a start from scratch run and it's not learning from its production traces." — Shubhankar Srivastava

- **Google Flights Breakdown:** Drag-and-drop time filters confuse computer-use agents that rely on standard click/type primitives. - **OpenTable Breakdown:** Common anti-bot defenses instantly block generic headless browser sessions and naive agent scripts. - **Core Diagnosis:** Agents act as "geniuses that never learn" because every single execution starts completely from zero.

---

[06:17](https://www.youtube.com/watch?v=p7gaxmCUiIU&t=377s) The AutoBrowse Architecture: Doer and Teacher Agents

To solve this amnesia problem without model fine-tuning, Shubhankar draws inspiration from Andrej Karpathy's "auto-research" loop concept—an autonomous experimentation framework that repeatedly tackles verifiable problems, inspects execution traces, and hill-climbs toward better performance. Adapting this concept to web navigation yielded the "AutoBrowse" architecture, built using two paired Claude Agent SDK agents: a "doer" agent and a "teacher" agent.

```mermaid flowchart TD accTitle: AutoBrowse Dual-Agent Hill-Climbing Architecture accDescr: The doer executes browser actions, while the teacher evaluates execution logs and writes updated instructions into a strategy file for the next iteration.

subgraph Execution ["Execution Stage"] A["Doer Agent (Claude SDK)"] -->|"Drives Browser via CLI"| B["Remote Browser (Browserbase)"] B -->|"Produces Screenshots & Logs"| C["Execution Traces"] end

subgraph Evaluation ["Evaluation & Learning Stage"] C -->|"Analyzes Failures & Hits"| D["Teacher Agent (Claude SDK)"] D -->|"Writes Lessons & Selectors"| E["strategy.md"] E -->|"Supplies Prior Context"| A end

subgraph Graduation ["Final Output"] E -.->|"Converges on Max Tricks"| F["skill.md (Reusable Skill)"] end ```

The doer agent's sole responsibility is driving the browser via the Browse CLI, issuing page interactions, clicking elements, and capturing accessibility trees and screenshots. Once the doer completes or fails an attempt, the teacher agent analyzes the resulting logs, screenshots, and reasoning traces to diagnose what worked and what stalled. Crucially, the system does not impose arbitrary constraints such as strict loop counts, token caps, or hardcoded timeouts; it embraces Rich Sutton's "Bitter Lesson" by allowing compute and iterative trial-and-error to discover optimal paths.

The bridge connecting individual runs is an evolving Markdown file titled `strategy.md`. When the teacher spots an error—such as an agent fruitlessly clicking an inaccessible calendar widget—it writes specific instructions and site-specific insights into `strategy.md`. The subsequent attempt begins with this document loaded directly into context, ensuring the next agent starts with cumulative operational knowledge rather than a blank slate.

> "After every iteration, the teacher thinks about what happened, looks at the screenshot, looks at the logs, and then thinks, 'Oh, you did not click on this button. You should actually click on this button instead, and it puts that in the strategy.md file.'" — Shubhankar Srivastava

---

[07:13](https://www.youtube.com/watch?v=p7gaxmCUiIU&t=433s) Live Demonstration: Discovering URL Shortcuts on Google Flights

Shubhankar launches a live demonstration of AutoBrowse in action. He supplies the system with a multi-constraint prompt: navigate to Google Flights and find a morning business class flight from San Francisco (SFO) to New York (NYC) for the upcoming Saturday, traveling with one carry-on bag. A remote Browserbase cloud session spins up on screen, with real-time reasoning traces showing the doer agent attempting to click through individual form controls and apply the business class dropdown filter.

During the first iteration, the agent stumbles and fails to execute the sequence efficiently. However, the teacher agent reviews the trace and uncovers a non-obvious shortcut: instead of clicking through fragile interactive calendar widgets and form fields, Google Flights encodes flight parameters directly into URL query strings. For iteration two, the teacher instructs the doer to bypass form navigation entirely and construct a direct query parameter URL. Shubhankar points out that while URL manipulation seems obvious in hindsight, human scriptwriters rarely take the time to map out full URL schemas manually.

As the loop continues through subsequent iterations, the teacher refines interactions, JavaScript snippets, and selector strategies until performance converges. Once the strategy stabilizes, the accumulated knowledge in `strategy.md` graduates into a permanent, reusable `skill.md` file. This finalized skill artifact serves as an optimized, deterministic blueprint that future production agents can execute immediately without repeating the trial-and-error discovery phase.

> "The agent was smart enough in this instance to think, oh, I'm wasting so many tokens and I'm wasting so much time clicking around where I could just use Google Flight's query parameter." — Shubhankar Srivastava

- **Live Flight Query:** SFO to NYC, upcoming Saturday departure, business class cabin, morning departure window, 1 carry-on bag. - **Iteration 1 Insight:** Scraping and clicking date pickers wastes excessive tokens and execution time; constructing URL query parameters directly avoids fragile UI elements entirely. - **End Product:** Conversion of `strategy.md` into `skill.md`, packaging verified selectors, navigation shortcuts, and custom JavaScript workarounds for future agents.

[11:26](https://www.youtube.com/watch?v=p7gaxmCUiIU&t=686s) Persisting Browser Tricks in the Strategy File

Shubhankar Srivastava details the concrete instructions that start accumulating in the strategy file after an agent runs into friction. Instead of wasting tokens and turns, the file explicitly encodes tactical workarounds: mandatory pre-filled URLs so the agent avoids clicking through calendar UI elements, warnings about known calendar reopen bugs, and instructions to inject JavaScript directly to drag a time-slider rather than attempting clumsy mouse-drag operations. By persisting these lessons across runs, future iterations avoid starting from scratch, resulting in massive gains in execution speed, reliability, and cost.

Returning to the live flight search run, Shubhankar notes that iteration two passed successfully. A real-time chart generated during the execution shows an immediate 4x drop in cost: while the initial baseline attempt cost $12, iteration two dropped to $3, despite running heavyweight Claude 3 Opus models for both the doer and the teacher. Iteration three arrives shortly after, showing fewer turns and lower latency, but it registers a failure due to an overcorrection. Shubhankar explains that this bounce is normal—iteration four is expected to ingest the failure, adjust the rules, and stabilize.

- **Persisted Tactical Lessons**: - Require pre-filled URLs instead of clicking calendar buttons. - Avoid known calendar reopen bugs. - Use JavaScript injection instead of mouse actions to move time-slider handles. - **Iteration Economics**: Cost fell from $12 in run 1 to $3 in run 2 (a 4x cost reduction) while using Claude 3 Opus for both the doer and teacher agents.

```mermaid flowchart TD accTitle: Strategy File Iteration Loop accDescr: The cycle of doer execution, trace evaluation by teacher, strategy updates, and cost reductions.

subgraph Execution["Execution Loop"] D["Doer Agent (Opus)"] -->|"Runs browser tasks"| T["Traces & Screenshots"] T -->|"Analyzed by"| TE["Teacher Agent (Opus)"] TE -->|"Writes tactics & fixes"| S["Strategy File"] S -->|"Guides next run"| D end

subgraph Metrics["Performance Trajectory"] M1["Run 1: $12 (Baseline)"] --> M2["Run 2: $3 (4x Savings)"] M2 --> M3["Run 3: Overcorrection Failure"] M3 --> M4["Run 4: Self-Correction"] end

S -.-> Metrics ```

---

[13:06](https://www.youtube.com/watch?v=p7gaxmCUiIU&t=786s) Conference Schedules and Circumventing OpenTable Blocks

Shubhankar walks through earlier test runs conducted prior to the presentation to show how the loop operates on varied web targets. In one example, the agent was tasked with navigating the AI Engineer conference schedule to identify talks and workshops discussing "hill climbing." Across successive iterations, replays captured via Browserbase showed cost improvements between 4x and 7.4x, even though every single run passed. The resulting skill distilled precise navigational rules: navigate directly to `/schedule`, do not retry blindly on errors, insert search keywords into the session filter input, add explicit brief waits to let dynamic content render, and parse the resulting session cards.

He then presents a more striking run on OpenTable, which initially threw an immediate `Access Denied` blocker on turn one. The teacher agent diagnosed that the block was an environmental anti-bot measure triggered by visiting certain route structures. By iteration two, the agent uncovered a URL shortcut: passing explicit query parameters (including latitude, longitude, and metro IDs) completely bypassed the bot-detection gates. The strategy codified explicit negative rules: never navigate directly to `/r` (which triggers access denied) and do not reload `/s`. Running with these parameters dropped task latency from 189 seconds down to 27 seconds, and reduced cost from $6.22 to just $0.51.

Reflecting on how the model bypassed defensive web infrastructure through trial and error, Shubhankar remarks: “Now these are the kind of learnings where a human could never have done but you just sort of embrace the bitter lesson and a let agents do the job and they will find these tricks for you.”

- **Conference Schedule Task**: Achieved up to 7.4x cost reductions across runs by learning to bypass homepages, wait for dynamic card hydration, and inject search keywords directly into session filters. - **OpenTable Bot Evasion**: - Initial error: Immediate `Access Denied` on turn one. - Workaround: Construct targeted search URLs using explicit query parameters (`latitude`, `longitude`, `metroId`). - Strict negative constraints: Never call `/r` or reload `/s`. - Efficiency delta: Execution time dropped from 189s to 27s; task cost plunged from $6.22 to $0.51.

---

[15:48](https://www.youtube.com/watch?v=p7gaxmCUiIU&t=948s) Production Reliability, Graduation, and the browse.sh Catalog

Shubhankar highlights that browser reliability is the primary roadblock for enterprise deployments. From a sales perspective, customers will not buy until an agent executes deterministically in production. Across client workloads, the AutoResearch loop routinely drives cost reductions ranging from 3x to 16x. Returning to the live flight demo, the teacher marks the skill as fully graduated. The agent runs end-to-end without further exploration, retrieving the requested one-way business class flight from San Francisco to New York for July 4th (8:00 AM departure, $649, with carry-on included). Shubhankar describes seeing the agent reverse-engineer complex web workflows as an absolute turning point: “When I first started looking at the the results here this was like my moment of like this is the mythos moment for browser agents you know like just reverse engineering the hell out of things.”

Instead of having human engineers spend hundreds of hours manually scripting interactions and scraping logic for individual targets, customers are embedding these loops into automated CI/CD pipelines. When production websites change and an agent breaks, self-healing loops trigger automatically, generate patches, and open pull requests against the skill repository.

To scale this beyond bespoke scripts, Shubhankar introduces `browse.sh` alongside an open-source `browse` CLI. Framing it as the modern “Yellow Pages for browser agents,” Browserbase has run these training loops across hundreds of popular web domains (such as Amazon, AllTrails, and Weather.gov), absorbing all model token costs so engineers can pull pre-baked skills directly into their agent architectures. Users can also enter any URL or workflow into browse.sh to asynchronously generate and receive a custom skill via email.

- **Customer ROI**: Production workloads consistently exhibit 3x to 16x reductions in execution cost once skills converge. - **Automated Skill Maintenance**: Broken production tasks trigger self-healing loops that patch skills and open automated pull requests without developer intervention. - **browse.sh Platform**: A public catalog of pre-trained web execution skills, accessible through the `browse` CLI or an online skill generator.

---

[20:28](https://www.youtube.com/watch?v=p7gaxmCUiIU&t=1228s) Extending Hill-Climbing to Coding, Research, and Verifiable Domains

Shubhankar argues that the pattern of persisting execution strategies across iterations applies far beyond web browsers. He points to common friction points in other agent domains: - **Coding Agents**: Frequently waste context and cycles relearning repository layouts, project conventions, and linting rules on every interaction instead of updating persistent instruction files like `AGENTS.md` or `CLAUDE.md`. - **Deep Research Agents**: Internal research agents repeatedly re-discover sources, pulling from Wikipedia while ignoring required company-preferred databases because they fail to retain navigational traces from previous runs. - **Internal Ops**: Enterprise workflows navigate internal dashboards and databases blindly because agents lack persistent memory of which tools or endpoints are functional and preferred.

He emphasizes that successfully porting the AutoBrowse methodology to any other domain relies strictly on two non-negotiable architectural requirements: 1. **A Replayable Environment**: A sandbox or runtime that can be reset and run repeatedly across multiple attempts. 2. **A Verifiable Outcome**: A quantifiable metric or deterministic signal that confirms whether an iteration succeeded or failed.

An audience member asks how an operator can trust that the AutoResearch loop is genuinely converging on valid behavior rather than hallucinating success. Shubhankar acknowledges the challenge of verification outside of pure machine learning research: “For Karpathy, it was quite easy because he was looking at the log loss of the experiment when he was training neural networks. Right? So it's very easy to see whether your log loss is going down or up.” In non-deterministic environments like web navigation, establishing reliable ground-truth verifiers is the central design problem.

[23:20](https://www.youtube.com/watch?v=p7gaxmCUiIU&t=1400s) Loop Design, Failure Resiliency, and Grindable Environments

Shubhankar Srivastava begins by contrasting simple optimization targets with the realities of web automation. In synthetic benchmarks or clean machine-learning runs, tracking whether loss has decreased is mathematically straightforward. In the messy world of browser agents, success criteria are rarely clean binary flags: knowing whether a flight ticket was successfully purchased might be clear-cut, but determining whether an agent found the "best flight options" is inherently fuzzy and subjective.

Addressing an audience question about whether a single mistake during the autoresearch loop could derail the entire optimization run, Shubhankar explains that non-monotonic progress is completely normal. Even after an agent achieves a successful run, subsequent iterations can and do fail. The secret lies in designing a resilient loop harness that does not overindex on transient mistakes or suffer from the machine-learning equivalent of exploding gradients. Borrowing an insight from Andrej Karpathy, Shubhankar notes that auto-research can be applied recursively: one can perform auto-research on the auto-research architectures themselves to discover whether a doer-teacher pair, a broader multi-agent harness, or an alternative setup works best across different problem domains.

```mermaid flowchart TD accTitle: Resilient AutoBrowse Optimization Loop accDescr: The loop alternates between a Doer agent operating in the environment and a Teacher agent extracting learnings to update strategy files. subgraph Run["Execution Phase"] A["Doer Agent"] -->|"interacts via browser"| B["Replayable Environment"] B -->|"produces logs, snapshots & traces"| C["Run Results & Failures"] end subgraph Learn["Refinement Phase"] C -->|"inspects traces"| D["Teacher Agent"] D -->|"extracts lessons / ignores blips"| E["strategy.mmd / SKILL.md"] E -->|"seeds next run"| A end ```

Beyond quantifiable metrics and verifiable outcomes, Shubhankar highlights a frequently overlooked ingredient: creating a "grindable" and replayable environment. Citing an idea discussed with Darko, he emphasizes that agents must be able to fail repeatedly and grind through iterations to fix themselves. Coding agents have a natural advantage because they can effortlessly snapshot state and replay execution using Docker containers or clean Amazon EC2 instances. Knowledge work, by contrast, is plagued by messy human-in-the-loop dependencies that make rewinding to an exact point in time difficult.

- Verifiable outcomes and quantifiable metrics are foundational, but an environment that cannot be reset easily cripples iteration. - Coding tasks excel because environments can be checkpointed and restored from a specific Docker image or EC2 state. - In domains where automated resets are hard, investing heavily into the replay environment yields massive returns for agent self-correction.

> “If your agent cannot grind through failures, it will be difficult for it to go for multiple iterations and fix itself.” — Shubhankar Srivastava

---

[26:52](https://www.youtube.com/watch?v=p7gaxmCUiIU&t=1612s) Open-Source Artifacts, Convergence Costs, and Browserbase Infrastructure

Transitioning to practical tooling, Shubhankar shares QR codes linking to the public AutoBrowse GitHub repository and the reusable skill definition on skills.sh. Using Claude Code or Codex, developers can invoke `/auto browse` directly, configure the models (such as running Claude 3.5 Sonnet or Opus 4.8 across the teacher and doer roles), and generate a browser skill automatically.

Addressing audience questions on Slido regarding how to optimize for convergence and manage iteration expenses, Shubhankar reframes the economics of LLM evaluation. While developers often worry that long iterative loops will become prohibitively expensive, empirical testing shows that running a cheaper model like Claude 3.5 Sonnet over more iterations often achieves the same final quality as a smaller number of expensive Opus runs, while remaining cheaper overall. A typical complex AutoBrowse generation run costs between $30 and $40, reaching $100 only in extreme edge cases—an upfront cost amortized over thousands of reliable production executions.

When asked how the concept transitioned from Karpathy’s autoresearch into AutoBrowse, Shubhankar recalls his experience working with Browserbase customers:

> “The way I solve it for them is I go into their office, I sit with them very forward-deployed, and I try to sort of just brute force and reverse engineer this. This is what I do. So I am the teacher agent. Can you automate me out of the process and create a loop borrowing from Karpathy and build the same thing?” — Shubhankar Srivastava

He adds that infrastructure directly dictates agent success in adversarial environments. Running on Browserbase enables agents to log into authenticated services, persist session cookies across runs, and integrate residential IP proxies to bypass CAPTCHAs and anti-bot protections without forcing the agent to solve login flows on every single attempt.

- AutoBrowse repository and skills are available via GitHub and skills.sh for Claude Code/Codex integration. - Cost tradeoffs show that running Claude 3.5 Sonnet across higher iteration counts often outperforms fewer runs of Opus at a lower total cost. - Complete AutoBrowse loops generally cost between $30 and $40 (capping around $100 for very intricate workflows), amortized over production usage. - Browserbase platform capabilities—session persistence, saved cookies, and residential proxy pools—shield the agent from repeated authentication friction and anti-bot defenses.

---

[30:56](https://www.youtube.com/watch?v=p7gaxmCUiIU&t=1856s) Code Anatomy: Evaluating Traces and Stepping Through Strategy Files

Shubhankar dives into the codebase, walking through `evaluate.mjs` to demonstrate how the teacher-doer iteration functions step-by-step. The process is strictly sequential: the doer runs a browsing session, after which the teacher analyzes the execution logs, UI snapshots, and interaction traces to write fresh learnings into a persistent `strategy.mmd` file.

Examining a real Google Flights execution run, Shubhankar points out the progressive refinements captured by the teacher: 1. **Iteration 1:** Discovers initial page mechanics and baseline search flow. 2. **Iteration 2:** Explicitly notes that dates must follow a strict ISO format. 3. **Iteration 3:** Flags that the agent wasted three unnecessary turns opening a bare page, reading the current date, and manually clicking the UI calendar, instructing the doer's executor to compute target dates programmatically instead.

He then pulls up an OpenTable reservation task targeting "The Big B's" restaurant. At turn one, the doer issues a `browse_open` command. At turn two, it executes `browse_snapshot`, which dumps the accessibility tree rather than raw HTML. Shubhankar explains that while this machine-readable accessibility tree looks like dense code to a human, the teacher agent ingests it alongside step-by-step screenshots, determines what went right or wrong, and outputs a clean markdown summary that updates `strategy.mmd`.

```mermaid flowchart LR accTitle: OpenTable Trace Analysis Pipeline accDescr: Step-by-step trace capture where browser actions produce accessibility trees and screenshots that feed into teacher summaries. subgraph Browser["Browser Turn Execution"] T1["Turn 1: browse_open"] --> T2["Turn 2: browse_snapshot"] T2 --> OUT["Accessibility Tree & Screenshot"] end subgraph Analysis["Teacher Synthesis"] OUT --> TRACE["evaluate.mjs Trace Evaluator"] TRACE --> SUM["Iteration Summary"] SUM --> MMD["strategy.mmd Update"] end ```

- `evaluate.mjs` drives the core loop, alternating between doer execution and teacher critique. - Traces capture granular accessibility trees (`browse_snapshot`) and screenshots rather than unstructured DOM dumps. - `strategy.mmd` functions as the persistent memory, accumulating step-by-step execution rules derived from past turn failures.

---

[33:54](https://www.youtube.com/watch?v=p7gaxmCUiIU&t=2034s) Task-Specific vs. Universal Skills and the Role of Harness Engineering

An audience member asks whether practitioners should build fine-grained, dedicated skills for individual tasks or attempt to train a single, monolithic, general-purpose browser skill. Shubhankar argues that agent reliability is fundamentally a software harness engineering problem rather than a model constraint. Whether one chooses specialized skills or a universal skill depends on user requirements and rigorous evaluation across candidate strategies.

To illustrate, Shubhankar uses the challenge of booking a dinner date. A developer could build separate, highly optimized skills for OpenTable, Resy, and Google Reservations. A well-engineered harness can interpret the high-level intent, dynamically summon all three platform skills, and orchestrate them to secure the best reservation:

> “The harness should be able to figure out which tools to use to get the job done. That’s like how I would go about it: figure out what are the most common use cases that your customers have, for those use cases what’s the best skill files, prompts, tools... and eval across all of these strategies and let the harness figure it out.” — Shubhankar Srivastava

When an audience member points out the philosophical parallel between this automated prompt-and-skill refinement loop and Stanford's DSPy framework, Shubhankar agrees enthusiastically. He observes that early declarative optimization frameworks like DSPy were years ahead of their time in treating prompt compilation and harness optimization as structured engineering disciplines.

- Reliability stems from harness architecture and evaluation rather than solely relying on larger foundation models. - Harnesses can dynamically compose narrow, high-reliability domain skills (e.g., OpenTable + Resy) to execute complex, ambiguous user goals (e.g., planning a date night). - The AutoBrowse iterative skill compilation approach shares foundational principles with early algorithmic prompt optimization frameworks like DSPy.

[36:15](https://www.youtube.com/watch?v=p7gaxmCUiIU&t=2175s) Verifiers, Skill Storage, and Cloud Deployment

Addressing questions around evaluation and optimization frameworks like DSPy, Shubhankar Srivastava highlights Browserbase’s ongoing research partnerships, including collaborative work with Microsoft on designing the right verifiers for autonomous agents. While running initial evaluations is often straightforward, teams quickly hit a meta-level dilemma regarding evaluation reliability and labor costs. To address this, Browserbase co-authored research exploring whether expensive human verification can be substituted with robust, automated verifier agents without compromising benchmark quality. Srivastava offers to link these research papers in the project's GitHub repository for practitioners looking to formalize their validation pipelines.

```mermaid flowchart LR accTitle: Skill Storage and Deployment Flow accDescr: Pipeline showing local markdown skill files deploying to production agent loops or Vercel Blob storage for browse.sh. subgraph Storage ["Storage Layer"] A["Local /skills Folder"] --> B["Vercel Blob Storage (browse.sh)"] end subgraph Consumption ["Agent Execution"] B --> C["On-Demand Agent Retrieval"] A --> D["Production Agent Loop"] end ```

Moving to how generated browsing skills are stored and served at scale, Srivastava shares his screen to show his local development setup. In the repository, each skill is represented as an individual Markdown file within a local directory—including a recipe searcher he built the evening before when planning dinner. For production systems, these files can be checked directly into version control and baked into the agent loop. In public multi-tenant environments like Browserbase's `browse.sh` catalog, the skills are stored in Vercel Blob storage, allowing agents to retrieve and invoke specialized site procedures on demand over the cloud.

- Running evaluations appears simple until teams face the recursion problem: “I think running evals is straightforward until you start questioning okay how should I eval my eval right like do I need to put all these humans to verify all these runs or is there a good verifier that I can build here and all of that uses the learnings from these papers as well.” - Browserbase published joint research with Microsoft exploring automated agent verification methodologies. - The reference AutoBrowse repository persists generated skills as plain Markdown files in a local `/skills` directory. - Production and public deployments (such as `browse.sh`) host skill files on Vercel Blob storage for dynamic, on-demand agent consumption.

[37:54](https://www.youtube.com/watch?v=p7gaxmCUiIU&t=2274s) Website Drift and Failure-Triggered Repair Loops

An audience member asks how teams can monitor websites that update their markup or layouts frequently and keep existing browsing skills from breaking. Srivastava explains that continuously running the full auto-research loop on a daily cron schedule burns an unnecessary amount of tokens on sites that have not changed. Instead, the preferred production strategy relies on reactive self-healing triggered directly by runtime errors. When an agent experiences an execution failure in production, the failure event automatically initiates an AutoBrowse research loop to diagnose the UI drift, regenerate the strategy, and open a pull request with the updated skill for human review.

For teams requiring tighter guarantees, Srivastava describes a hybrid schedule where lightweight sample runs execute every two or three hours to test skill integrity without full retraining. Deciding between proactive cron jobs and reactive repair loops comes down to cost and accuracy trade-offs. Working directly with enterprise customers on this problem, Srivastava points out that real-world engineering constraints vary wildly: some organizations refuse to pay for continuous 100% synthetic uptime checks when an occasional failure-driven patch delivers a ten-fold reduction in operational costs.

- The optimal maintenance model balances proactive checks against reactive healing: “The current working theory that we have right now is you don't want to waste tokens running auto research every day.” - When an agent encounters an unrecoverable failure in production, it triggers an AutoBrowse loop that generates a pull request containing the updated skill for an engineer to approve. - Periodic health checks (e.g., executing test runs every two to three hours) offer an intermediate trade-off between constant polling and purely reactive repairs. - Enterprise priorities often diverge from benchmark goals; many production teams prefer a 10x cost reduction over maintaining 100% continuous validation accuracy across every workflow.

[39:26](https://www.youtube.com/watch?v=p7gaxmCUiIU&t=2366s) Optimization Objectives and WebMCP Programmability

When asked about the criteria the teacher agent uses beyond basic task completion, Srivastava emphasizes that the auto-research loop's objective function is completely configurable. While some implementations stop as soon as a task succeeds, production teams frequently optimize for latency or context efficiency. In latency-sensitive real-time applications, the loop aggressively prunes navigational overhead. In fact, Srivastava notes that teacher agents sometimes discover they do not need browser automation at all: the doer reverse-engineers the site's internal network requests, allowing the final skill to fire raw HTTP API calls directly and complete the task in under 500 milliseconds.

```mermaid flowchart TD accTitle: AutoBrowse Optimization Modes accDescr: Three paths of optimization an AutoBrowse teacher agent can target: pure reliability, execution speed, or minimal context turns. A["AutoBrowse Objective"] --> B["Pure Reliability"] A --> C["Latency Reduction"] A --> D["Context Efficiency"] B --> E["Iterate until stable success regardless of cost"] C --> F["Bypass DOM: Reverse-engineer backend API (500ms)"] D --> G["Minimize conversational turns & token footprint"] ```

Srivastava addresses whether the loop can leverage emerging browser standards like WebMCP (Web Model Context Protocol), a mechanism recently integrated into Google Chrome that lets websites expose programmable actions—such as booking interfaces—directly to agents. Browserbase recently added WebMCP support to its SDKs and published tools for webmasters, though broad consumer site adoption remains early. When evaluating task success, Srivastava relies on crisp binary outcomes (such as finding the cheapest Google Flights itinerary or securing an OpenTable reservation) and avoids heavily engineered constraints.

- Teacher agents can discover shortcuts that circumvent browser rendering entirely: “What I've seen sometimes these auto research loops are so smart they realize that browsers are not needed at all and they just reverse engineer the API and they fire the API and that's the fastest that you'll ever get to.” - AutoBrowse optimization profiles include pure reliability (iterating until confident), context minimization (reducing interaction turns), and raw speed (reaching sub-second runtimes). - WebMCP standardizes browser-level programmability, allowing websites to expose native structured actions directly to browser agents. - Browserbase shipped WebMCP capabilities in its core SDK and built webmaster deployment tooling, though live web adoption remains in its infancy. - Adhering to the "Bitter Lesson," Srivastava prefers minimal guardrails and simple binary success evaluations over heavily hand-crafted heuristic rules.

[42:38](https://www.youtube.com/watch?v=p7gaxmCUiIU&t=2558s) Real-World Environments vs. Mocked Side Effects and CAPTCHAs

An audience question from Ravish asks whether Browserbase has considered offering a "verifier as a service" that mocks downstream side effects, such as completing checkout transactions on platforms like Amazon. Srivastava clarifies that while synthetic reinforcement learning environments (like mocked Salesforce or SAP instances) are popular for academic benchmarks, Browserbase intentionally avoids synthetic mocks in favor of interacting directly with live, production websites. By handling proxies, fingerprinting, and session infrastructure at scale, Browserbase focuses on providing agents with reliable access to the real web rather than simulated sandboxes.

Turning to security obstacles, Srivastava answers an inquiry about whether advanced edge challenges and CAPTCHAs (like Google reCAPTCHA) consume large token budgets or impede generated skills. He explains that CAPTCHA handling does not burden the agent's language model because it is resolved entirely at the infrastructure layer. Browserbase intercepts and solves these challenges natively within custom Chromium binaries, ensuring that agents and skills do not waste reasoning cycles or context tokens attempting to parse security puzzles visually.

- Browserbase prioritizes direct production execution over synthetic mocks: “We try to rely more on real world websites.” - While enterprise workflows on platforms like Salesforce and SAP are often trained in simulated sandboxes, Browserbase designs its stack to operate on live portals. - CAPTCHA handling requires zero LLM token consumption: “We don't really need to use an AI to solve a CAPTCHA. It's solved at the Chrome binary layer.” - Infrastructure-level interception solves bot challenges out of the box before DOM states are passed to the agent.

[45:19](https://www.youtube.com/watch?v=p7gaxmCUiIU&t=2719s) Evaluation Mechanics, Traces, and Skill Portability

Srivastava explains how the teacher evaluates the doer's performance across iterations. During every run, the system creates a dedicated iteration folder containing structured JSON trace logs. The teacher agent, powered by the Claude Agent SDK, reads these execution traces alongside the global system prompt and the active `strategy.md` file. By comparing the intended goal against the exact browser actions recorded in the trace, the teacher identifies points of failure or inefficiency and edits the strategy file to guide the subsequent iteration until the skill reaches a stable graduation threshold.

```mermaid flowchart LR accTitle: Teacher Evaluation and Skill Graduation Loop accDescr: The iterative feedback loop where doer traces inform teacher updates until graduating to an agnostic Markdown skill. subgraph Iteration ["Evaluation Cycle"] A["Doer Run Traces (JSON)"] --> B["Teacher Agent (Claude Agent SDK)"] B --> C["Update strategy.md"] C --> A end subgraph Output ["Graduation"] C -->|"Converged"| D["Universal Skill (.md)"] D --> E["Claude Code / Stagehand / Custom SDKs"] end ```

Responding to a live question regarding platform lock-in, Srivastava demonstrates that graduated skills are universally portable. Because a skill is simply a human-readable Markdown file detailing navigational steps, URL structures, and DOM strategies, it can run in any agent framework. He demonstrates running a skill inside Anthropic's Claude Code CLI to locate San Francisco events without relying on Browserbase or its Stagehand automation framework, underscoring that the output artifact remains independent of any specific infrastructure provider.

- The teacher refines execution strategies by analyzing structured run outputs: “It reads the traces, it has the system prompt, it has the previous strategy, and it comes up with a better strategy.” - Execution data is captured in per-iteration folders storing JSON traces that record every network call, DOM interaction, and error state. - Graduated skills are completely decoupled from Browserbase tooling: “It need not use Browserbase. It need not use Stagehand. It is essentially a skill. It is a portable artifact you can use in any agent or any SDK.” - Srivastava validated portability by feeding an exported skill directly to Claude Code to plan an itinerary for upcoming San Francisco events.

[47:30](https://www.youtube.com/watch?v=p7gaxmCUiIU&t=2850s) Layered Defenses Against Prompt Injection and Malicious Web Content

The session concludes with an in-depth discussion on defending browser agents against prompt injection attacks, where malicious websites embed hidden adversarial text designed to poison the auto-research loop or hijack production agents. Srivastava outlines a multi-tiered defense architecture that starts at the network perimeter. Browserbase enforces an infrastructure feature called Safe Mode, which restricts outgoing network traffic via firewalls, limits agents strictly to whitelisted domain boundaries, and restricts the types of actionable DOM events agents can execute on untrusted portals.

To counter subtler threats—such as an attacker embedding hidden prompt injection payloads inside a public Amazon product listing—Browserbase implements an automated audit phase for newly synthesized strategies. Before any skill is published to the public `browse.sh` registry, it must pass an independent security reviewer agent that inspects the generated Markdown for unexpected redirections, exfiltration attempts, or unsafe execution patterns. Srivastava strongly advises developers building consumer-facing automation to maintain comprehensive prompt injection guardrails around all inputs, untrusted page content, and output actions.

- Mitigating adversarial injection requires hardware and network controls before the model ever reads page content: “At the infrastructure layer, at the network firewall, we control things.” - Browserbase Safe Mode provides strict domain whitelisting, preventing agents from being lured to unauthorized third-party URLs. - Generated skill files undergo automated auditing by a security reviewer agent before being released into public catalogs. - Unstructured consumer portals (e.g., e-commerce marketplaces containing untrusted merchant text) represent high-risk attack surfaces that require strict input/output guardrails.

[49:05](https://www.youtube.com/watch?v=p7gaxmCUiIU&t=2945s) Layered Guardrails and the Threat Vector of the Open Web

Addressing the severe security hazards inherent in autonomous web browsing, Shubhankar emphasizes that protecting agents requires active collaboration with customers to implement robust guardrails before any instruction ever reaches the execution harness. He articulates a defense-in-depth model structured across three distinct tiers. The first defense sits at the entry point of the surface area, validating and sanitizing queries before they touch the agent. The second defense operates at the skill layer, ensuring that the synthesized strategy cannot be hijacked or tricked by adversarial text, hidden prompts, or untrusted DOM content encountered across arbitrary web pages. The third defense sits at the underlying infrastructure layer, imposing hard network restrictions and fine-grained action permissions.

At the infrastructure level, developers can establish granular privilege policies analogous to the approval mechanisms found in developer tools like Codex or Claude Code—such as setting operations to always allow, allow once, or require explicit confirmation for specific mutating actions. He notes that Browserbase offers this type of protection out of the box through Safe Mode, which prevents unconstrained access to sensitive vectors. Recognizing the open internet as an uniquely volatile operating environment, Shubhankar warns developers never to underestimate the attack surface of automated browsing: “I know browser is a crazy threat vector when you think about it like it has access to anything on the wild web.” He stresses that teams must treat this reality with the utmost urgency: “Uh and take that super seriously from a prompt injection POV.”

```mermaid flowchart TD accTitle: Three-Layer Browser Agent Security Architecture accDescr: Diagram showing user query entering input guardrails, passing to the skill layer, and executing under infrastructure sandbox controls. subgraph InputTier ["1. Surface Entry Tier"] Query["Incoming User Instruction"] --> PreHarness["Input Guardrails & Validation"] end subgraph SkillTier ["2. Skill Strategy Tier"] PreHarness --> SkillExec["Hardened Skill Instructions"] SkillExec --> DOMSanitize["Adversarial Prompt & DOM Shielding"] end subgraph InfraTier ["3. Infrastructure & Runtime Tier"] DOMSanitize --> SafeMode["Browserbase Safe Mode Sandbox"] SafeMode --> NetControl["Strict Network Allow/Deny Lists"] SafeMode --> ActionControl["Action Restrictions (Always / Once / Deny)"] end ```

- **Entry Surface Controls:** Rigorous pre-harness query filtering designed to prevent adversarial instructions from initiating untrusted runs. - **Skill-Level Hardening:** Strategy logic crafted to withstand indirect prompt injection buried in dynamic page text and malicious page elements. - **Infrastructure Sandboxing:** Browserbase Safe Mode enforces network boundaries and configurable action approval tiers (allow once, always allow, or block).

---

[50:33](https://www.youtube.com/watch?v=p7gaxmCUiIU&t=3033s) Monitoring Website Drift and Triggering Self-Healing Loops

As the Q&A session nears its conclusion, an audience question explores how teams should handle operational drift—specifically when target websites change their DOM markup, introduce layout redesigns, or upgrade anti-bot detection systems. Shubhankar explains that long-term skill maintenance relies on automated monitoring paired with event-driven repair routines. In his own workflows, he leverages Anthropic managed agents connected to scheduled cron jobs that test production skills proactively rather than waiting for user-facing breakage.

By running these jobs on an ongoing cadence—such as executing a designated browsing skill daily at 9:00 AM—the team establishes an automated health check. If the scheduled task completes successfully, the system logs the run and continues as normal. If the task fails due to a layout change or an anti-bot obstacle, the failure dispatches an automated notification that immediately kicks off an iterative healing loop. Under this healing regime, the teacher-doer architecture re-inspects the failed run's artifacts and replays, diagnoses the new obstacles, and updates the skill strategy file to restore reliability. Shubhankar explains: “What you could do is like I used uh Anthropic managed agents quite a bit. Uh so I would run a cron job on managed agents where I would say use this skill every day at 9:00 am to do this job and if this is a success all good if this is a failure send me a notification here that will trigger a healing loop uh so that's how I sort of can bake in uh checks and balances around websites changing antibots changing etc.” Concluding the talk, he flashes the repository QR code back on screen and invites attendees to continue the conversation in person at the Browserbase conference booth.

- **Scheduled Drift Probing:** Daily cron execution (e.g., Anthropic managed agents running tasks at 9:00 AM) verifies that target website selectors and flows remain functional. - **Automated Healing Loops:** Run failures trigger immediate alerts that feed failure traces, screenshots, and logs back into the teacher agent to repair the skill file. - **Resilience Against Bot Mitigations:** Systematic monitoring isolates failures caused by anti-bot changes or layout shifts without demanding manual code rewrites from scratch.

## Notable quotes > [23:20](https://www.youtube.com/watch?v=p7gaxmCUiIU&t=1400s) If your agent cannot grind through failures, it will be difficult for it to go for multiple iterations and fix itself. — Shubhankar Srivastava

> [02:41](https://www.youtube.com/watch?v=p7gaxmCUiIU&t=161s) Essentially, there's no learning happening. Every agent run is a start from scratch run and it's not learning from its production traces. — Shubhankar Srivastava

> [42:38](https://www.youtube.com/watch?v=p7gaxmCUiIU&t=2558s) We don't really need to use an AI to solve a CAPTCHA. It's solved at the Chrome binary layer. — Shubhankar Srivastava

> [33:54](https://www.youtube.com/watch?v=p7gaxmCUiIU&t=2034s) The harness should be able to figure out which tools to use to get the job done. — Shubhankar Srivastava

## Mentioned Shubhankar Srivastava, Browserbase, Claude, Claude Opus, Claude 3 Opus, Claude 3.5 Sonnet, Claude Code, Claude Agent SDK, Andrej Karpathy, Rich Sutton, Google Flights, OpenTable, Resy, Google Reservations, Amazon, AllTrails, Weather.gov, Salesforce, SAP, Microsoft, Stanford, DSPy, Vercel Blob, WebMCP, Google Chrome, Chromium, Stagehand, browse.sh, Browse CLI, AutoBrowse, EC2, Docker

## Video description An agent succeeds at a flight search, then fails on its next iteration. Shubhankar Srivastava uses that uneven progress to show how browser agents can learn a task without changing model weights. His AutoBrowse loop pairs a doer that operates the browser with a teacher that reads screenshots, traces and results after each run. The teacher writes lessons into a strategy file, and the next attempt starts with those lessons instead of starting from zero. The flight demo discovers that a URL can encode the search directly, avoiding repeated calendar clicks and an awkward time filter.

The final artifact is a skill file that future agents can reuse. Earlier runs on a conference schedule and restaurant search show how the file preserves navigation choices, waiting behavior and site specific constraints, while replays make the claimed results inspectable. Srivastava then explains what another domain would need to use the same approach: an environment that can be replayed and an outcome that can be verified. Questions expose the limits of that requirement, especially when finding an acceptable flight is easier to judge than finding the best one. The discussion explores task specific versus general skills, different optimization goals, iteration costs and how to measure the verifier itself. For changing websites, he describes monitoring and failure triggered repair loops that produce changes for human review. Prompt injection adds another constraint: generated skills need scrutiny alongside controls on inputs, allowed actions and destinations.

Speaker info: - https://shubhankar.xyz/ - https://github.com/shubh24/ai-engineer-autobrowse

Timestamps: 0:00 - Improve the skill instead of model weights 1:39 - Browser tools and reliability 3:30 - Time filters, blocks, and repeated failures 5:07 - Why each run starts from zero 6:17 - Borrow the autoresearch loop 7:13 - Live Google Flights task 8:08 - Doer and teacher architecture 9:32 - Persist lessons in a strategy file 9:59 - A failed run discovers a URL shortcut 11:13 - Graduate the strategy into a reusable skill 12:07 - Success followed by another failure 13:06 - Conference schedule and restaurant examples 16:46 - Inspect the completed flight search skill 18:20 - A catalog of browsing skills 20:28 - Apply the approach to other domains 21:39 - Replayable environments and verifiable outcomes 22:39 - Who verifies the loop's judgments? 25:40 - Invest in the replay environment 27:41 - Convergence, iteration costs, and infrastructure 30:56 - Read the implementation and traces 33:54 - One skill per task or a general skill? 36:31 - Verifiers, storage, and skill maintenance 39:26 - Choose an optimization objective 41:06 - WebMCP and outcome questions 45:09 - Evaluation, graduation, and portability 47:30 - Prompt injection and layered controls 50:33 - Monitor drift and trigger repairs

---

Watch along with the corrected transcript, summary, timestamps and resources on this talk's official AIE page: https://ai.engineer/talks/p7gaxmCUiIU-youtube-b87810eee05e8ebc7f29

Join our upcoming AI Engineer conferences (see https://ai.engineer landing page):

- AIEi Shanghai, November 5–6 — the first for meeting top Chinese labs and engineers - AIE CODE in SF, November 10–12 — the top AI Coding conference returns - AIEi Sydney, December 7–8 — the first AIEi run alongside NeurIPS 2026 in Sydney

---

[Source](https://www.youtube.com/watch?v=p7gaxmCUiIU)

Reading summary

Autonomous browser agents frequently fail in production not from a lack of model intelligence, but because they start each task from scratch without episodic memory. Shubhankar Srivastava of Browserbase argues that iterative 'doer-teacher' optimization loops can reverse-engineer web interactions, compiling production execution traces into durable, low-cost skills without altering model weights. The Amnesia Problem in Web Automation (2:41) Modern browser agents driven by frontier models struggle with reliability, frequently getting trapped in interaction loops, failing on dynamic UI elements like sliders, or getting halted by anti-bot measures such as OpenTable's 'Access Denied' barriers. Fine-tuning models directly on individual websites is unscalable because web markup changes constantly and targets proliferate continuously. Srivastava diagnoses the fundamental issue as episodic amnesia: agents behave as 'geniuses that never learn' because every run starts from zero. Making agents dependable in commercial settings requires external scaffolding, execution skills, and tooling that persistently capture tactical lessons from production traces rather than relying on raw model intelligence alone. Dual-Agent Hill Climbing and Strategy Graduation (6:17) To address this execution gap, Browserbase developed AutoBrowse, adapting Andrej Karpathy's autonomous experimentation concept. The architecture pairs two agents: a 'doer' that executes browser actions and dumps accessibility trees or screenshots, and a 'teacher' that critiques the resulting traces. When failures occur, the teacher writes specific interaction workarounds and selectors into a persistent `strategy.md` file, providing cumulative operational knowledge for the next attempt. Over successive iterations, the teacher refines navigation instructions until performance converges, graduating the strategy into a permanent, reusable `skill.md` artifact. In a live Google Flights demonstration, this feedback loop autonomously realized that clicking fragile interactive calendar widgets was inefficient, discovering instead that it could directly construct URL query parameters to set travel dates and cabins. Economic Gains and Latency Optimization (11:26) By substituting brittle browser navigation with compiled procedural shortcuts, the optimization loop dramatically slashes execution overhead. In test runs on Google Flights, iteration costs fell fourfold from an initial baseline of $12 down to $3. On OpenTable, the teacher bypassed bot-detection gates by encoding specific latitude, longitude, and metro parameters directly into the query, reducing execution time from 189 seconds to 27 seconds and cost from $6.22 to $0.51. Srivastava notes that customer production workloads routinely see cost reductions ranging between 3x and 16x once skills converge. Furthermore, the teacher agent's objective function can be tuned for latency: in some instances, loops have reverse-engineered internal network requests to bypass DOM rendering entirely, executing tasks via raw HTTP API calls in under 500 milliseconds. Infrastructure Prerequisites and Adversarial Defenses (23:20) Extending this iterative refinement to other domains requires two non-negotiable architectural foundations: a replayable, grindable environment that can easily be reset, and a verifiable outcome metric. While coding agents benefit from clean container snapshots, live web environments demand heavy infrastructure support, including residential proxies, session cookie persistence, and binary-level CAPTCHA solving that does not burn language model tokens. Operating across the open web also exposes agents to indirect prompt injection embedded in untrusted web pages. Srivastava advocates for a three-tier defense-in-depth model: pre-harness query validation, skill-level hardening against adversarial DOM content, and infrastructure controls like Browserbase's Safe Mode to enforce network firewalls, domain whitelisting, and strict action approval tiers. Maintenance, Portability, and Website Drift (37:54) Because web layouts and anti-bot systems evolve, production skills require automated maintenance. Rather than burning tokens by running expensive daily optimization loops, Browserbase implements reactive self-healing: execution failures in production automatically trigger an AutoBrowse loop that diagnoses UI drift and generates a pull request with an updated skill. Alternatively, teams run scheduled lightweight cron jobs to probe skill integrity. Graduated skills remain portable across frameworks. Because the output artifact is a plain Markdown file detailing interaction logic, it can be hosted in cloud blob storage, pulled from registries like `browse.sh`, or fed directly into external runtimes like Claude Code without requiring proprietary SDK locks. While the AutoBrowse framework demonstrates that external harnesses can solve agent amnesia and drive 3x to 16x cost reductions, establishing reliable automated verifiers remains a central hurdle when tasks involve subjective outcomes rather than clean binary criteria.

Source quotes

[23:20] Shubhankar Srivastava: If your agent cannot grind through failures, it will be difficult for it to go for multiple iterations and fix itself. [2:41] Shubhankar Srivastava: Essentially, there's no learning happening. Every agent run is a start from scratch run and it's not learning from its production traces. [42:38] Shubhankar Srivastava: We don't really need to use an AI to solve a CAPTCHA. It's solved at the Chrome binary layer. [33:54] Shubhankar Srivastava: The harness should be able to figure out which tools to use to get the job done.

Original transcript

[0:00] All [0:12] right. Hello. Thank you everybody for [0:15] being here. Uh I know it's the last [0:17] workshop of the day. I appreciate all of [0:19] you turning up here. U I am show uh I'm [0:23] here to talk about hill climbing skills. [0:26] uh specifically how can you improve your [0:29] agents without really going into the [0:31] models without really going in the [0:33] weights. Uh I have a slido uh up here as [0:36] well. I'll just uh flash the QR code in [0:40] case any of you want to uh put in any [0:42] questions and we can address them [0:43] towards the end as well. U a little bit [0:46] about me. I'm Sh. I'm uh on the sales [0:49] engineering team at browserbase. Uh I [0:52] essentially work with our customers [0:55] deploying browser agents in production. [0:58] Uh I'm not sure maybe we'll do a show of [1:00] hands. Uh how many of you know what [1:03] browser agents are? [1:06] Anybody? All right, a few. Amazing. Uh [1:10] have any of you built browser agents [1:12] before? Have any of you used these [1:14] tools? Show hands. Okay, a few of them. [1:18] Amazing. Nice. uh maybe I'll I'll talk a [1:22] bit about what browser agents are before [1:24] diving into the whole concept of hill [1:26] climbing for skills uh so that it [1:29] becomes clear what we're doing towards [1:30] the end of the talk we'll also look into [1:33] how you can do this in your domain in [1:35] your uh field as well. [1:39] All right. Uh [1:41] so quick primer on what browser agents [1:44] are. If you have an agent that needs to [1:47] drive a browser, you basically call that [1:49] a browser agent, right? Browser becomes [1:51] the tool to your agent. And how do you [1:54] control it? You control it with some [1:56] sort of CLI, some sort of MCP, some sort [1:59] of method where you can control the [2:02] browser and say, "Hey, go and click on [2:04] this red button. Go and click on the [2:06] refresh button. Go and uh take a [2:09] screenshot or take a snapshot of the [2:12] accessibility tree of this website." [2:14] Right? So these are this low-level tools [2:16] that your agent can have access to to [2:18] control the browser. Uh and browserbase [2:22] is essentially a platform for you to [2:24] build and run these browser agents. We [2:27] provide the infrastructure and the SDKs, [2:30] the skills, the CLI that go with [2:33] building these browser agents. Uh and [2:35] and we're going to be using all of these [2:37] tools to see what hill climbing does [2:39] look like. [2:41] All right. Uh I want to talk a bit about [2:46] the biggest problem with agents today [2:49] essentially reliability right uh people [2:52] are frustrated with browser agents [2:55] getting stuck or getting into an [2:57] anti-border capture issue uh or maybe [3:00] just forgetting or being lazy and not [3:02] completing the task. Uh and we see [3:05] reliability essentially is an [3:07] engineering problem. I think the recent [3:09] models the Opus and the Fable when we [3:11] when it came out for those three days it [3:13] was really really smart right uh but we [3:16] see that a lot of the work that has to [3:19] be done for reliability is the hardness [3:22] around it is the tools that you give it [3:24] it is the skills that you give here u [3:27] and and that's what we're going to be [3:28] sort of looking at how do we improve the [3:30] reliability of these browser agents I [3:33] wanted to give you all a few examples of [3:35] like where these agents suck today, [3:38] right? If I zoom in a bit, you'll see [3:40] this browser agent is trying to like go [3:43] on Google flights and click on the uh [3:47] it's trying to drag the time filter and [3:49] it's unable to figure out how do I drag [3:51] that. Dragging is a tricky operation for [3:54] these agents to do and it's getting [3:56] super confused. It doesn't know what the [3:57] HTML element for that is, what the [3:59] JavaScript for that is. It's not really [4:01] able to do this and it gets stuck and [4:03] it's not able to book my flight tickets. [4:05] Uh, another example I wanted to show uh [4:09] is a browser agent on open table. Let's [4:11] say I have my open claw or my Hermes [4:14] agent and I need to book a date night [4:16] and I tell hey go to open table and and [4:18] find me some and and look at the [4:20] anti-bot that it runs into, right? Uh, [4:22] and it's just like access denied. I [4:24] don't know what happens here. Uh, but [4:26] like can my browser agent get around [4:28] these issues? Can they be more reliable? [4:30] And can they book my date night [4:32] restaurants is the biggest question I [4:33] wanted to ask here. uh and this is where [4:38] we sort of look at different ideas to [4:41] improve the reliability of these agents [4:44] and one of them uh oh actually let's [4:47] let's dig into the root cause here right [4:49] like why this is happening u we have [4:52] like these opus models these five five [4:54] the super nice general purpose you might [4:57] have used codeex computer use very nice [4:59] but when you want to sort of do uh [5:01] longunning [5:04] tricky websites like the open tables or [5:07] the Google flights it sort of becomes [5:10] very difficult to run these in [5:11] production and these sightspecific [5:13] tricks or these javascript that you need [5:15] to run on your browser the agent doesn't [5:17] know about them right and the most [5:19] important thing is it's every run starts [5:22] from zero you're not really having any [5:24] sort of learning baked into your runs I [5:27] think Darkh talked about this a couple [5:29] days ago it's like why is progress on [5:31] computer use still so slow today. Uh [5:34] although computer use seems like a [5:36] verifiable problem, why is it still not [5:39] solved? Essentially, there's no learning [5:42] happening. Every agent run is a start [5:45] from scratch run and it's not learning [5:48] from its production traces. U and these [5:50] browser agents, I call them geniuses [5:52] that never learn. Um and some people [5:54] might say, oh, why don't you fine-tune [5:56] on something some like open table or [5:58] restaurant websites. Uh but it it's not [6:01] really scalable, right? uh if you're [6:03] thinking of building agents, you can't [6:04] really be fine-tuning for every new task [6:06] that comes up. You want some you want a [6:10] a modality where it can learn from its [6:12] production traces uh and get better over [6:14] time. [6:17] This is where we sort of started [6:18] thinking about and this was like six [6:20] four to six months ago where Karpadi [6:22] came up with this idea of auto research. [6:25] This is essentially a loop, right? Uh [6:28] and this loop sort of takes a verifiable [6:32] problem statement and goes at it, tries [6:34] it again and again, looks at its own [6:37] traces and tries to improve itself. And [6:40] we wondered how can we apply this to [6:42] browser agents? How can we apply this uh [6:45] to these complicated workflows, these [6:47] complicated websites which are not [6:49] deterministic, they change every day, [6:51] they have different captures every time. [6:53] Uh how can we learn from these? Right? [6:56] Uh this is where I want to sort of [6:59] switch into a demo and sort of show this [7:01] live. Uh and I'm going to start with uh [7:08] a prompt over here. So I want to maybe [7:11] do something like uh go to Google [7:13] flights and uh find me a business class [7:18] flight from SF to New York for this [7:21] Saturday. I want to have one carry-on [7:24] bag. Uh and this flight should be in the [7:27] morning. [7:29] I hope my Okay, this is correct. [7:35] Just paste this and run this. Right. So [7:38] starting a browser based session here. [7:40] Uh so what you see in the center is a [7:43] remote browser has just spun up and has [7:45] navigated to Google flights. If I expand [7:48] my agent here, uh it's doing all of [7:52] these things. This is the reasoning [7:54] traces that you can see live and it's [7:55] trying to sort of apply the business [7:58] class filter. So it it'll do its job and [8:00] and and we'll come back to this. Uh but [8:02] I wanted to talk about what's really [8:04] happening here. Uh u and the [8:08] architecture behind this auto research [8:10] loop. [8:15] So we we fired the Google flights task. [8:18] Uh and and and the architecture here is [8:21] this auto browse loop that we call this. [8:24] We took auto research and we took [8:25] browsing. We call it auto browse. And [8:27] this loop essentially is two clawed [8:29] agent SDK loop uh agents. Uh you can [8:33] call one as the doer agent and one as [8:35] the teacher agent. This doer agent its [8:38] job is to actually go and do the [8:39] browsing. It has access to the browse [8:41] CLI that we talked about. It has access [8:44] to the browser uh and it goes and sort [8:46] of clicks on buttons, looks at the [8:49] screenshot, tries to get me information [8:51] and the teacher teacher's job is to look [8:54] at the traces of the doer and learn what [8:56] went wrong and what went right and they [8:59] keep working together [9:01] until they find a very good solution [9:04] that converges. Right? Uh we don't tell [9:07] it how many loops you need to go for. We [9:10] don't tell it what's the success factor [9:13] success uh rate that we want. We don't [9:16] tell it hey we need to do this within x [9:18] number of tokens or x number of seconds. [9:21] We just give it a broad prompt and we [9:23] let the bitter lesson we embrace the [9:25] bitter lesson and we're like okay let [9:27] the doer and teacher go at it. Uh and [9:30] the key thing that happens is there is a [9:32] strategy.md file. uh that is essentially [9:35] the artifact across multiple runs. [9:39] After every iteration, the teacher [9:42] thinks about what happened, looks at the [9:44] screenshot, looks at the logs, and then [9:46] thinks, "Oh, you did not click on this [9:48] button. U you should actually click on [9:51] this button instead, and it puts that in [9:53] the strategy.mmd file." And that [9:55] artifact carries over to the next [9:56] iteration, and it improves from there. [9:59] Maybe we'll go back to our demo and see [10:02] what happened. All right. So, iteration [10:04] number one, uh, there was a failure [10:06] here. Uh, and the learning from this [10:09] iteration was skip the entire form [10:12] calendar widget because you can just do [10:15] a query parameter and you don't really [10:17] need to click around, right? The agent [10:19] was smart enough in this instance to [10:21] think, oh, I'm wasting so many tokens [10:23] and I'm wasting so much time clicking [10:26] around where I could just use Google [10:27] Flight's query parameter. Now this is [10:30] obvious in hindsight but maybe as a [10:32] human I wouldn't have thought of this [10:34] and I wouldn't have written this [10:35] complicated query parameter myself right [10:38] uh so this is the learning from the [10:41] first iteration now it's going and doing [10:43] things in iteration number two it'll [10:45] keep going on till it feels like it has [10:48] maxed out all the tricks it has maxed [10:50] out all JavaScript hacks that it could [10:52] do uh and and and then uh think about [10:57] okay this is the skill that I can use [10:59] from here. [11:03] I think the key the key thing uh that I [11:06] wanted to also talk about is at the end [11:09] of this converged loop the output that I [11:13] want is a skill.mmd file. This skill.md [11:16] file would be what I give to my future [11:19] agents. Uh and it would have a [11:22] collection of all the hacks or all the [11:24] tricks or all the learnings from [11:26] previous runs. We'll we'll look at what [11:28] that scale might look like. Uh it could [11:31] look like, hey, a pre-filled URL is [11:34] mandatory. Don't go around and waste [11:36] your clicks and tokens clicking on these [11:39] buttons. There is a reopen bug in the [11:42] calendar. Don't do this. Uh if you want [11:45] to move the drag uh if you want to drag [11:48] your time stamp thing, uh use JavaScript [11:51] injection. Don't try to do it with the [11:53] mouse. Right? These are the kind of [11:55] tricks that are now persisted for every [11:58] future run and gives me so many more [12:01] savings uh from a performance, [12:03] reliability and cost perspective. [12:07] Now uh we'll go back to our uh run and [12:12] see if it has passed. Okay, so iteration [12:14] number two has passed and if you can see [12:16] uh the savings in in cost are like 4x. [12:21] Uh this chart is being created on the [12:23] fly. Uh so it started from $12 and the [12:26] second one was $3. And keep in mind my [12:29] agents here are Opus 48 both for the [12:32] doer and uh for the uh teacher. [12:37] Now iteration three just came in. Uh the [12:40] number of turns decreased. The latency [12:42] is also better. Uh but there was a [12:45] failure in iteration number three. [12:47] there's something that it sort of [12:48] overcorrected on and iteration number [12:50] four should probably be learning from [12:52] that and we'll see if it does that. [12:55] Uh while it's doing its own thing, I [12:58] wanted to show some previous runs that I [12:59] did uh just in preparation for this uh [13:02] demo. U so for example, I went to uh [13:06] auto browse and I was like hey go to the [13:08] AI engineers website and tell me which [13:10] talks and workshops are talking about [13:11] hill climbing, right? uh and it did that [13:14] across multiple iterations and you can [13:16] see the replay of each of these. These [13:18] are replays that are coming from [13:19] browserbase super helpful important for [13:22] uh observability perspective and as you [13:24] can see uh it sort of uh got better in [13:28] ter in terms of cost 7.4x 4x uh even [13:32] though it was able to pass in all the [13:35] iterations, I still got improvements in [13:37] cost. Now I wanted to actually see what [13:39] are the learnings from this skill, [13:41] right? Uh how do you sort of uh persist [13:45] all of these learnings? It says go [13:47] straight to the shed schedule page. If [13:50] there are open errors, don't try retry [13:52] blindly. Run this. Uh type the keyword [13:55] into the session search input. Wait [13:58] briefly. So like have these weights [14:00] because sometimes websites take a lot of [14:01] time to load. So you you can have these [14:03] kind of tricks uh and then read the [14:05] results. Uh you know like there there's [14:08] a lot of learning that is happening in [14:10] these skills. Maybe I wanted to show [14:12] some interesting ones. I wanted to show [14:15] this open table one because I started [14:17] with this [14:19] blocker of like access denied when we [14:21] started this demo. Uh and if we go to [14:23] the uh screenshot, this was the access [14:27] denied thing that we got right in in [14:29] iteration number one. Let's see how it [14:31] learned from there. Uh it realized that [14:34] the uh block is environmental because [14:36] the website doesn't want this agent to [14:39] do something. uh in iteration number [14:41] two, it figured that if I do a query [14:44] parameter and if I put a latitude [14:46] longitude metro ID, I automatically get [14:49] that uh UI that works. Do not open /r [14:54] that's blocked and do not reload / s. [14:57] These are the places where the blocks [14:59] kick in. If you don't do that and you [15:01] follow this technique instead, your [15:03] agent will work, right? Uh and as you [15:06] can see, uh it sort of went from $6.22 [15:09] 22 to 0.51. [15:12] The time taken went from $189 seconds to [15:15] 27 seconds. Uh and the scale it came up [15:18] with was here. Uh build the search URL, [15:22] do the date math like this. Uh this is [15:26] the known metro pins for San Francisco. [15:29] Uh and and follow these steps, right? [15:31] Never call / r because that will return [15:34] access denied. Now these are the kind of [15:36] learnings where a human could never have [15:38] done but you just sort of embrace the [15:40] bitter lesson and a let agents do the [15:42] job uh and they will find these uh [15:44] tricks for you. [15:48] I want to uh also show uh a few examples [15:52] of like the kind of improvements we see. [15:54] Uh the reason why this problem is so [15:56] close to my heart is because I from a [15:59] sales perspective am deploying these [16:02] browser agents into production. uh the [16:06] sale is only done when the browser agent [16:08] is successful. So reliability is like [16:10] the most important thing for me and this [16:12] is what I this is the key tool that I [16:14] take to my customers and I'm like just [16:16] fire your agents let them do this auto [16:19] research thing and your agents will be [16:21] successful in production. Uh and these [16:23] are the kind of u success metrics that [16:25] people are looking for up to 16x cheaper [16:28] 12x cheaper 3x cheaper uh on different [16:32] tasks. Of course, every task is [16:33] different. The bottlenecks on each task [16:35] are different. So, uh you might see [16:37] variance there. But the idea is you can [16:40] hill climb on these uh techniques. [16:46] I wanted to go back to our live demo and [16:48] see if there's any uh improvement. All [16:53] right. So, we have a graduated skill u [16:56] and the skill applied end to end. No [16:58] further learning. The results are [17:00] showing correctly. one-way business [17:01] class, San Francisco to New York, July [17:04] the 4th, one carry-on bag included. Uh, [17:06] and there is 8 a.m. $649. Right? So, it [17:10] was able to get all of this and I can [17:11] sort of watch the entire replay to see [17:14] if that is accurate. Uh, it was able to [17:16] sort of figure out this entire skill and [17:19] the fast path is encode the whole query [17:21] in the URL. strongly preferred the again [17:25] like I know I'm I'm showing these things [17:27] but when I first started looking at the [17:30] the results here this was like my moment [17:32] of like this is the mythos moment for [17:35] browser agents you know like just [17:36] reverse engineering the hell out of [17:38] things uh I I look at this with some of [17:40] my customers the customers are [17:41] mind-blown uh earlier they would be [17:45] writing scripts for each of their [17:48] websites they would be like hey I need [17:50] to spend so many engineering hours on [17:53] like just reversing these websites. How [17:56] can I scale this to 100 a thousand [17:58] websites and now our customers have [18:00] actually built these entire pipelines of [18:03] uh these skills that get generated on [18:06] the fly pull requests that get created [18:08] automatically. If there is a failure in [18:11] production, an agent kicks in and starts [18:13] healing the skill as well. So there's a [18:15] lot of uh engineering that we can do [18:18] around this to make leverage to leverage [18:20] this technique. [18:22] Uh I wanted to also show what we've done [18:25] with this right we actually went ahead [18:28] and created this product called [18:29] browse.sh [18:31] uh we were like if we can do this for [18:34] one website we can do this for five why [18:36] not do this for like 500 websites and [18:38] give it for free for anybody to do this. [18:40] Uh and like if you remember in the 19 or [18:44] wherever like in the '9s before internet [18:46] you had the yellow pages uh you had [18:49] details about accessing people, [18:52] businesses etc. This is yellow pages for [18:56] browser agents right it has details [18:58] about how to do any task on any website [19:01] on the web. U I wanted to actually show [19:03] this to you all. uh [19:07] you can actually install browse CLI and [19:10] use all of these skills completely for [19:13] free. We've eaten up the token costs [19:15] ourselves and we're making this [19:17] available for you all. Uh so if I go to [19:20] let's say search products on Amazon uh I [19:23] can see the entire scale is over here. [19:26] When should you use the scale? How [19:28] should you look at like pricing and [19:31] availability and whatever right? uh this [19:33] is how you use this lot of details about [19:36] Amazon the screenshots u and I just give [19:40] this browse skills CLI to my openclaw [19:44] and it just knows when to pull the [19:46] alltrails.com skill when to pull the [19:49] weather.gov skill and and this makes my [19:52] life so much easier. uh we essentially [19:54] are running the same auto research loop [19:56] behind the scenes uh and making this uh [20:00] available for you all. In in fact if you [20:02] would want to you can go here and you [20:04] can be like hey go and make a skill for [20:07] me to uh look up this uh events in San [20:11] Francisco for this week. Uh you do this [20:15] generate a skill and this will sort of [20:17] do this for you. This will send you an [20:19] email and you can then use your skill uh [20:21] in whatever agent you want to, right? Uh [20:24] so we've done this for browser agents. [20:28] Uh and I wanted to talk about like how [20:30] can you potentially do this for any [20:33] other domain, right? Uh I think coding [20:37] agents is like one place where I have [20:39] seen this frustration where the agent is [20:42] relearning the repo conventions every [20:45] time. it's not applying the linting [20:47] rules correctly and I have to tell it [20:48] every time and maybe you can have your [20:50] cloud MD and agents MD and improve that [20:53] but the the fundamental premise is like [20:56] the agent should not be relearning your [21:00] tasks from zero right uh I've seen this [21:04] in research agents we run a bunch of [21:06] like deep research agents for our [21:08] internal workflows uh and every time it [21:10] struggles and it tries to relearn oh I [21:14] pulled this information from Wikipedia [21:15] IA uh but I did not pull it from this [21:19] other website. Uh I was like why not? Uh [21:22] like we always do this other website you [21:25] should learn from your previous traces. [21:27] Uh and you should pull information. Uh [21:30] internal ops you have all these [21:31] dashboards all these databases all these [21:33] internal tools but your agents don't [21:35] know which ones work and which ones are [21:37] preferred. Ideally you should be [21:39] applying this all research technique. Uh [21:42] a and the two conditions that are [21:45] required is that a you should have a [21:48] replayable environment of some sorts so [21:50] that you can run these loops and b you [21:52] should have some sort of verifiable [21:54] outcome some sort of quant quantifiable [21:56] metric for you to run these loops. Yes, [21:58] I see a question there. [22:18] I realized that [22:39] Yeah. [22:43] >> Yeah. [22:44] >> Yeah. Exactly. Uh so I think the [22:46] question is like how do you know that [22:49] the auto research loop is accurate [22:51] whether it is converging and when it [22:53] says it's converging it's actually [22:54] converging. When it says it's good it's [22:57] actually good. When it says it's bad [22:58] it's actually bad. like how do I have [23:00] confidence uh on what the agent is [23:03] saying and what the loop is doing. I [23:05] think that's where I want to sort of [23:07] talk about the verifiable outcome here. [23:09] Uh for Karpathi, it was quite easy [23:11] because he was looking at the log log [23:14] loss of the experiment when he was [23:16] training neural networks. Right? So it's [23:18] very easy to see whether your log loss [23:20] has gone down or not. Uh but in many [23:22] real world scenarios that is not the [23:24] case for browser agents. As I explained, [23:26] uh, whether you book a flight ticket or [23:29] not, that might be straightforward, but [23:32] did you find the best flight options for [23:34] me might be slightly fuzzy uh and might [23:38] not be the best uh [24:05] Yeah, if if I understand this correctly, [24:06] you're saying like one mistake in the [24:09] loop can sort of throw off the entire [24:11] auto research run. Uh and how do you [24:14] prevent against that? I I I have [24:17] actually seen where some uh iterations [24:20] actually fail. uh for example if we uh [24:24] look at the previous slide [24:29] uh there's like failures happening after [24:32] there have been successes right uh so [24:34] it's not always that once you succeed [24:37] all the future iterations will always be [24:38] successful but I think it's the design [24:41] of your loop where you can make it [24:43] resilient to u failures uh where it [24:47] doesn't overindex on those uh and still [24:50] sort of correct protects itself uh from [24:53] any mistake that happened in previous [24:54] runs. Uh I think we're trying to sort of [24:56] prevent like uh what do they call like [25:00] exploding sort of loss sort of [25:02] scenarios. Uh I think you could do that. [25:05] Uh in fact Karpathi talks about this [25:08] like this is one way of doing auto [25:10] research. You can actually auto research [25:12] on different ways of doing auto research [25:13] itself. Uh it's like is the teacher doer [25:16] loop the best? Should there be a [25:18] multi-agent architecture to do this or [25:20] is there any other approach that might [25:22] actually work better for research tasks [25:24] in neural networks versus browser [25:26] agents? Right? You could have different [25:29] designs of your loops to prevent [25:31] situations like those. Uh and I think [25:33] going back to my point, as long as you [25:36] have a verifiable outcome or some [25:38] quantifiable metric, I think you should [25:40] be able to design uh a good uh loop [25:44] here. uh replayable environment I think [25:48] sometimes is under discussed uh but [25:51] extremely important. I think he was [25:53] talking about Darkh he said having a [25:55] verifiable outcome is important but also [25:57] you should have uh extremely grindable [26:00] environments or extremely grindable uh [26:04] use cases as well. Right? If your agent [26:07] cannot [26:09] uh grind through failures, it will be [26:11] difficult for it to uh go for multiple [26:15] iterations and fix itself. Uh so do you [26:18] have like coding agents is a good [26:20] example because you can start from a [26:22] docker image or you can start from uh a [26:25] particular EC2 and and you can replay [26:27] from different parts of the uh stack. Uh [26:30] but maybe uh knowledge work is probably [26:33] not that easy to replay. uh maybe [26:36] there's like a lot of different human in [26:38] the loop actions that make it very hard [26:39] to replay from a particular point in [26:41] time. So investing in your environment [26:44] also can be super uh rewarding in these [26:47] cases. [26:52] Amazing. Uh I just wanted to share the [26:56] uh URLs of uh the auto browse loop that [27:00] I uh demoed today. So if you scan the QR [27:03] you'll be able to pull the uh GitHub for [27:05] that. Uh and also the second QR here uh [27:08] is the skill itself. You can just do [27:11] /auto browse on your claude code or [27:13] codeex and be like hey /auto browse use [27:16] opus48 on both of these uh teacher and [27:19] doer and go create a skill for me to do [27:21] this. Uh [27:25] I'm looking at one of these questions. [27:28] Uh and also if any of you have ideas of [27:32] something we want to sort of put the [27:35] agent against feel free to type it in [27:37] the slido and we can actually try it [27:39] live. Uh but let's look at this [27:41] question. Is there a way to optimize for [27:42] convergence? Uh I think this is the same [27:44] question that we were just talking [27:45] about. Uh [27:48] high cost is interesting because you [27:50] might think that u doing longer [27:54] iterations might become costlier over [27:56] time. Uh I have run a bunch of evals on [27:59] this where I can throw a sonnet at this [28:03] problem but run it for longer iterations [28:05] versus throwing opus for lesser [28:07] iterations. I think you can get the same [28:10] job done by throwing sonnet larger [28:13] number of iterations but still cheaper [28:15] overall. Uh also this cost is amotized [28:18] right like one auto browse loop that we [28:21] probably run is like maybe $30 $40 if [28:23] it's super complicated $100 at most. Uh [28:26] so this is amotized as long as those [28:29] costs make sense for you uh to deploy [28:31] this in production uh just go for it. Uh [28:34] but there's a lot of interesting things [28:36] uh that you could do uh with evaling the [28:38] kind of loops that you're generating. Uh [28:41] yes I will share the slides as well. Uh [28:43] I can do this on the GitHub itself. [28:47] Uh [28:49] can you share how you thought about uh [28:53] auto research into auto browse? Uh I [28:55] mean the process was fairly [28:57] straightforward. Uh took the GitHub repo [29:01] that Karpathi put out uh and then [29:03] there's all this context about my [29:05] customers uh that I have uh where I was [29:08] like hey these customers face these [29:10] kinds of reliability problems uh the way [29:13] I solve it for them is I go into their [29:15] office I sit with them very forward [29:17] deployed uh and I try to sort of just [29:20] brute force and reverse engineer this. [29:22] This is what I do. So I am the teacher [29:24] agent. Uh can you automate me out of the [29:28] process and create a loop borrowing from [29:31] uh Karpathi and and and and build the [29:33] same thing? And of course there's a lot [29:34] of iterations on top of that. Uh there's [29:37] a lot of engineering as well uh to [29:39] simplify this. Uh but yes, that was the [29:43] process of converting this. [29:48] Can you use this in a proxy? Can you be [29:49] logged into a site? Yes. Uh again this [29:52] goes back to the environment that you [29:54] design. Uh browserbased the [29:55] infrastructure that all of this is [29:57] running on uh gives you the ability to [29:59] log into websites uh save the cookies uh [30:03] reuse those cookies in future runs so [30:05] you don't have to log in every time. Uh [30:07] makes it super easy for you to uh [30:09] replicate real world workflows. Uh and [30:13] yes you can use proxies as well. If you [30:15] have uh particular IP addresses, [30:18] residential IP addresses, you can bring [30:20] those in as well. Uh a lot of success is [30:24] definitely how good your infrastructure [30:26] and environment is, especially in these [30:28] adversarial scenarios, uh like captures [30:31] and antibots. Uh so making the [30:33] infrastructure as good as possible is [30:35] definitely of value. [30:42] Amazing. any any asks for something that [30:45] we might want to run live? Yes. [30:48] Question. [30:56] >> Great, great question. Actually, I [30:57] wanted to sort of show the code as well. [30:59] Uh so if you uh look at the GitHub, [31:02] you'll be able to find this evaluate.mjs [31:04] file. Uh [31:07] essentially, it lists out the entire [31:09] loop. Uh what's happening here? Uh but [31:11] to answer your question, the teacher and [31:14] the doer are u working one after the [31:20] other for every loop. So the teacher is [31:24] finding learnings in every iteration. Uh [31:27] maybe if I show the demo uh over here uh [31:31] let's go to the one we just did. [31:33] Iteration one had these learnings and [31:35] then iteration two had some other [31:37] learnings. Right? you should include the [31:40] explicit ISO date in this particular [31:42] format. Iteration three would have some [31:44] other learnings. The date resolution [31:46] still opens the bare page and clicks [31:48] wasting three turns uh reading today's [31:51] date uh or having the executor compute [31:54] today. So it's figuring out something [31:55] about the date time. So in every [31:57] iteration the teacher is learning [31:59] something new and putting this back into [32:02] the strategy.mmd. Maybe I'll make this [32:04] real by looking at the traces for one of [32:06] these runs. Uh [32:10] so [32:12] over here these are all my runs. Uh [32:16] so for example this was the open table [32:18] reservation example u and uh this was [32:22] the strategy.mmd that the teacher sort [32:24] of appended to every time. So uh step [32:27] zero you should do this so on and so [32:29] forth. Uh this was the task this was the [32:32] uh prompt that I gave it. Find me [32:34] reservations for the big B bigs [32:35] restaurant. uh and these were the kind [32:37] of traces that uh it was sort of [32:39] learning from. It would take screenshots [32:41] would take these sort of traces at every [32:44] step is doing browse open this is what [32:47] the output is. Then turn number two [32:50] browse snapshot. Snapshot essentially [32:52] takes the accessibility tree and this is [32:54] the snapshot. So it's reading this [32:56] entire thing and at every step thinking [32:59] of like what went right and what went [33:01] wrong. U yeah I mean this is like [33:03] machine code. uh I won't show this [33:06] entire thing. Uh but yeah, this is the [33:08] summary that it also writes uh at every [33:10] iteration. Uh so yeah, this there's a [33:12] lot lot of like engineering tweaks that [33:14] you can do to suit this to your domain [33:17] as well. U and uh like the easiest way [33:21] to uh read this is [33:25] we put it out here on skills.sh as well [33:29] uh in auto browse u and you can sort of [33:32] do something like this. [33:54] Great question. [34:03] Such a good question. I think that goes [34:05] back to like [34:07] how do you increase the reliability of [34:10] your uh agents. Uh reliability is an [34:14] engineering problem. Uh you can design [34:17] your harness to sort of go for one skill [34:20] for each task or you can be like hey one [34:23] skill for every task that you will ever [34:25] see. Uh I think it entirely depends how [34:28] you do the harness engineering entirely [34:31] depends on the use case that you have u [34:34] and you run evals with all these [34:36] different strategies uh and see which [34:38] one works for you. for us. This was [34:40] Yeah. [34:55] Yes. [35:00] >> Yeah. Yeah. And this is where again the [35:04] harness comes into the picture. If the [35:06] harness has the intelligence to bring in [35:09] skills from Open Table and from Resi and [35:12] Google res Google reservations, maybe it [35:16] can bring all of those in to figure out [35:18] what is the best way to book me a date [35:20] night. [35:21] That's your solution, right? Like the [35:23] harness should be able to uh figure out [35:26] which tools to use to get the job done. [35:29] That's like how I would go about it. uh [35:32] figure out what are the most common use [35:34] cases that your customers have for those [35:37] use cases. What's the best scale files, [35:40] prompts, tools, maybe you want to do [35:42] like specific tools for specific use [35:44] cases, right? Maybe you want to do [35:46] generic tools for all use cases. Like [35:48] that's the same argument here is like do [35:50] you do one skill for each or one skill [35:52] for all uh and eval across all of these [35:55] strategies and and let the hardness [35:57] figure it out. [36:01] Yes. [36:02] >> Yes. Yeah. I mean it's a very similar [36:04] concept as this, right? Uh so DSPY. [36:09] >> Yeah. I I think those guys were on to [36:11] something really cool like they were [36:12] around like 3 four years ago where they [36:15] started building DSP and Japa all of [36:18] that. I think uh we have been doing a [36:22] lot of research uh along these lines as [36:24] well. Browserbase works with uh [36:26] Microsoft. We actually released a paper [36:28] uh about how to build the right sort of [36:31] verifiers uh for your agents. Uh I think [36:34] running evals is straightforward until [36:36] you start questioning okay how should I [36:39] eval my eval right like [36:42] do I need to put all these humans to [36:44] verify all these runs or is there a good [36:46] verifier that I can build here and all [36:48] of that u uses the uh learnings from uh [36:52] these papers as well. I can share all of [36:55] that in the GitHub as well. [37:00] I'll also see if there's any new [37:02] questions here. [37:05] Where and how were these skills stored [37:07] retrieved at scale. Uh right now for [37:10] this repo uh all of these get stored in [37:12] my cloud over here. So each of these is [37:15] a scale file. This was my recipe [37:17] searcher. Uh I wanted to build some I [37:20] wanted to cook some dinner yesterday. So [37:22] I was like, "Hey, make me some recipes." [37:24] Uh, [37:26] so all of these are currently in my [37:28] local uh skills folder. Uh, but if you [37:31] want to ship this uh in production, of [37:34] course, you can uh bake this into your [37:37] agent loop. Uh, or if you want to I [37:40] think for browse.sh the public uh [37:43] publicly available uh skills that we [37:45] have, I think we use versel blob storage [37:47] to store all these skills uh and make [37:48] them on demand. [37:51] Hope that helps. Uh, [37:54] websites change frequently. What's your [37:55] STR strategy for monitoring that and [37:58] keeping the website skill updated? Yeah, [38:00] this is such a great question. Uh, and [38:03] this is what I was alluding to earlier [38:04] is like how do you keep these skills up [38:08] to date because the website's going to [38:09] change. The [38:13] the current working theory that we have [38:15] right now is you don't want to waste [38:17] tokens running auto research every day. [38:20] you want to probably trigger or if it's [38:23] failing at any particular point in [38:25] production, right? So, we build these [38:28] loops or like systems where as soon as [38:31] we run into any production failure, we [38:34] run an auto research loop to see what [38:36] went wrong to heal it, create a pull [38:38] request, that goes directly to a human [38:42] engineer and they fix it automatically. [38:44] The skill gets updated. Uh so yes there [38:47] is that notion of like production [38:49] failure when this might happen. Uh but [38:52] it's a trade-off. You can run an auto [38:54] research or you can run a sample run [38:57] every 2 hours every 3 hours to see if [39:00] the skill is working uh perfectly. So [39:03] there's different ways of having the uh [39:06] cron jobs uh check for the accuracy of [39:09] these skills. This is actually one thing [39:11] that I'm actively working on with the [39:13] customer right now to implement. And [39:15] there's all these questions, right? [39:16] People don't want 100% accuracy. People [39:19] might want like 10x cheaper cost. So [39:20] like uh that's the uh question that [39:23] comes up. What's your teachers criteria [39:26] here in the example? Separate from [39:28] reaching the goal seems to do some [39:30] optimizations for turns. What other [39:32] criteria do you use? Uh [39:35] so again this is the design of the auto [39:38] research loop. Uh there's one way of [39:41] telling your order research that hey I [39:43] want to focus on just success. As soon [39:46] as you get me success I'm fine just give [39:48] that to me. Uh that's typically not what [39:51] everybody wants to do. Uh people want to [39:53] minimize on uh latency. Uh because maybe [39:57] this is a real time run where you're [39:59] focused on how quickly can you get the [40:01] job done. U so if that is the optimizing [40:04] metric and you focus on getting it as [40:06] fast as possible. What I've seen [40:08] sometimes these auto research loops are [40:09] so smart they realize that browsers are [40:12] not needed at all and they just reverse [40:15] engineer the API and they fire the API [40:17] and that's the fastest that you'll ever [40:19] get to. Right? Uh so sometimes uh I've [40:22] seen situations where uh it finds a [40:25] solution that works in 500 milliseconds. [40:28] U [40:30] sometimes it is the number of turns, [40:31] right? you want to minimize your context [40:34] loop uh and you want to minimize the [40:36] number of uh turns it takes to complete [40:38] the task. Uh in my instance, I think [40:41] what I have done is it's your decision [40:44] to pass or converge whenever you want to [40:47] pause it. I'm not really concerned about [40:49] the costs. Uh I care most about [40:52] reliability. So keep doing it as long as [40:55] you feel it's relevant and then it's the [40:57] teacher agent's job to uh call it day [41:00] when it wants to. [41:03] But yeah, you can uh try all different [41:06] strategies. Uh if a website supports [41:09] WebMP, will this use it? WebMCP is so [41:11] cool. U it's essentially this concept. I [41:14] don't know if everybody knows this, but [41:16] it's this concept of like making your [41:20] website programmable. It's it's come out [41:22] in this new Google Chrome release [41:24] recently where you tell your u website [41:28] that if anybody wants to book a [41:30] reservation they can do xyz on the [41:33] website. It's kind of like an API but [41:35] it's on the browser. Uh right now I [41:39] don't think our oral research loop [41:40] supports this. Uh but I don't think it [41:44] would be uh too hard to do that. Uh in [41:48] fact we recently shipped support for [41:50] WebMCP with our SDKs that we launched. [41:53] Uh we also made some tools for website [41:56] owners to ship web MCPS at scale. Uh but [41:59] yeah this is an interesting one. We've [42:01] not really seen a lot of adoption with [42:02] Web MCP as yet. [42:06] Uh ground truth for pass. That's [42:10] interesting. Uh I think as long as the [42:12] job was done as long as for example in [42:14] Google flights uh if it found the [42:17] cheapest flights in open table if it [42:20] found a reservation uh so it it [42:22] evaluates itself. It's typically a [42:25] binary uh and yeah again I I I fully [42:30] embrace the bitter lesson. I try not to [42:32] put many guardrails in uh these [42:36] evaluations. [42:38] Uh, have you considered a verifier as a [42:41] service for browsing tasks, including [42:43] something that allows us to mock certain [42:44] side effects like purchases? [42:47] Uh, I might have to ask uh [42:52] a followup on that. What do you mean by [42:54] verifier as a service? Ravish, if you're [42:57] here. Yes. [43:29] Uh [43:32] so uh do you mean that [43:36] yeah if I understand this correctly like [43:38] you have like different environments [43:40] that you can use for RL training and you [43:42] want to like mock what happens if I am [43:45] replicating Amazon if I click on [43:47] purchase what happens after [43:50] Uh [43:54] no because uh we try to rely more on [43:58] real world websites. Uh browser base we [44:01] have the privilege of being able to [44:04] access most of the web with our uh [44:07] access to different websites uh and [44:09] capture solvers etc. Uh so we try to [44:13] mimic the real world as much as possible [44:15] but we've been seeing a lot of these RL [44:17] environments where people are mimicking [44:19] websites to train uh and build browser [44:21] agents that are uh really good at like [44:24] browsing Salesforce, really good at like [44:26] browsing SAP. Uh we try to do that on [44:30] real world portals as much as possible. [44:33] [snorts] [44:34] Are edge captures easy for the skill to [44:36] move past or is it token incentive? uh [44:39] what is the point of edge captures for [44:40] website owners? Edge capture if I'm [44:43] understanding this is like a particular [44:44] kind of capture. It's kind of like [44:46] Google recapture etc. uh [44:50] yeah I would probably talk about this [44:53] separately. Uh I don't think we can talk [44:56] about like what it what captures we can [44:57] solve or not. Uh but it's not a token [45:00] inensive job per se because it's an inf [45:03] like it's thing that's done at the [45:04] infrastructure layer, right? uh [45:09] we don't really need to use an AI to [45:10] solve a capture. Uh it's solved at the [45:13] Chrome binary layer. Uh and we do that [45:15] out of the box. [45:19] How does the teacher evaluate the doer [45:21] and when is the skill markers graduated? [45:23] Yes. So uh it evaluates the doer by [45:27] essentially reading these traces. Uh for [45:30] every iteration there is this folder [45:32] that gets created. uh there are these uh [45:35] JSON files. Uh it reads these uh it has [45:38] a clawed agent SDK loop uh and it [45:41] essentially figures out what the [45:43] learnings from that loop are. Uh [45:45] actually I think I have a slide that [45:48] talks about this at the end. [45:53] Yeah. Uh this is kind of like the prompt [45:55] for this uh where it says that you're a [45:58] browser automation agent. You can use [46:00] these tools uh and the strategy.md gets [46:04] injected at every iteration and this is [46:07] how the uh [46:10] uh doer the teacher evaluates the doer's [46:13] job. Uh so yeah it reads the traces it [46:15] has the system prompt it has the [46:17] previous strategy and it comes up with a [46:19] better strategy. [snorts] [46:27] Amazing. Uh [46:31] I hope we've covered all the questions. [46:34] Uh [46:36] any other questions live that we might [46:38] want to talk about? Yes. [46:44] >> Yeah, it can work with the stage hand [46:46] SDK. It can work in your cloud code. Uh [46:48] it's essentially a markdown, right? Uh I [46:50] took uh a skill and I told my clot code, [46:54] can you find events in SF for this [46:56] Friday, July the 3rd? and went on and [46:58] did this. So it it need not use browser [47:01] base. It need not use stage hand. It is [47:03] essentially a skill. It is a portable [47:04] artifact you can use in any agent or any [47:07] SDK. [47:30] Yeah. Yeah. Great question. I think if I [47:32] understand this correctly, how do you [47:34] prevent against prompt injection? Uh [47:37] because websites could trick my auto [47:41] research loop in learning something bad [47:44] that can do some malicious behavior in [47:46] in a production setting. Uh there's a [47:49] few ways we do that. One is at the [47:52] infrastructure layer itself. Uh we have [47:54] a bunch of uh [47:57] protections around our browsers itself. [48:00] Uh we have this browserbased safe mode [48:03] where you can sort of control which URLs [48:06] it can go to. You can control what kind [48:08] of actions it can take on different [48:10] websites. So at the infrastructure [48:12] layer, at the network firewall, we [48:14] control things. Now you might say that [48:16] it is possible that Amazon.com gets [48:18] hacked someday. How do you prevent [48:20] against like malicious actions on those [48:22] portals as well? Uh I think we also try [48:26] to uh run our skills that are generated [48:30] against some security uh reviewer agent [48:32] of sorts uh before we actually publish [48:35] this on browser.sh. Uh so that's like [48:37] one way of doing it. uh we encourage our [48:42] customers to have uh a good level of uh [48:46] prompt injection guardrail uh surface [48:48] area uh if they're sort of giving this [48:51] out to c consumers uh for example some [48:54] customers of ours uh might want to [48:56] automate something on Amazon and what if [48:58] somebody puts uh a listing on Amazon [49:02] which has some hidden instructions and [49:04] how do you prevent against those kind of [49:05] things uh and that's where uh we work [49:08] with our customers to build the right [49:09] kind of uh guardrails before the uh [49:13] instruction even reaches the hardness. [49:16] Um so yeah I think I would put like [49:18] three layers here. One is [49:21] as the as the query is even entering our [49:24] surface area we control that at the [49:26] skill layer we make sure that our skill [49:28] is not uh tricked and at the [49:31] infrastructure layer we have network [49:32] restrictions and action restrictions. [49:35] You can even do something like what [49:36] codeex or plot code does u like always [49:39] allow or allow once or allow any action. [49:42] Uh you can this is what like [49:44] browserbased safe mode offers out of the [49:46] box. I know browser is a crazy threat [49:48] vector when you think about it like it [49:50] has access to anything on the wild web. [49:53] Uh and take that super seriously from a [49:55] prompt injection POV. [50:02] Amazing. I'll wait for 10 more seconds [50:06] if there's any other questions. This was [50:09] fun. I really enjoyed this. I love the [50:11] kind of questions as well. Uh [50:15] and yeah, I will put the QR code back on [50:19] screen in case any of you want to grab [50:22] this. Yes. [50:33] Yeah. [50:44] >> Yeah. Yeah. Exactly. Yeah. I think we [50:46] talked about it a while back. What you [50:49] could do is like I used uh entropic [50:51] managed agents quite a bit. Uh so I [50:53] would run a chron job on managed agents [50:55] where I would say use this skill every [50:58] day at 9:00 am to do this job and if [51:01] this is a success all good if this is a [51:02] failure send me a notification here that [51:05] will trigger a healing loop uh so that's [51:07] how I sort of can bake in uh checks and [51:12] balances around websites changing [51:14] antibbots changing etc. [51:22] I'm going to hang around here. Uh also [51:25] uh I'll be at the browserbased booths [51:28] all days of the conference as well. So [51:30] uh in case you're curious and want to [51:32] chat more uh love to do that. Thank you. [51:50] >> [music]