Bright Data Releases Free AI Agent Proxy Tool Following Meta Legal Wins
Ai4
Anti-bot defenses systematically block autonomous AI agents at scale, requiring dedicated proxy and unlocking middleware to maintain real-time public web access.
## Background
This video was published on YouTube by the Ai4 channel on August 26, 2025. It features a talk by Ariel Shulman, Chief Product Officer at Bright Data, who discusses how to equip autonomous AI agents with real-time web access using advanced search, intelligent crawling, and multi-source integration while navigating ethical standards.
## TL;DR
Ariel Shulman explores the legal and technical hurdles AI agents face when accessing public web data, framing agents as proxies for human intent that are routinely blocked by anti-bot protections. He outlines how Bright Data fought landmark legal battles against tech giants like Meta and X to defend public web scraping rights. The presentation details Bright Data's infrastructure, including its 100M+ IP proxy network, Web Unlocker, dynamic CAPTCHA bypassing, and automated web scrapers that convert raw HTML into token-efficient structured data. Finally, Shulman demonstrates a live side-by-side comparison using Claude Desktop with Bright Data's Model Context Protocol (MCP) server, announcing a free tier for developers to connect agents directly to real-time web capabilities.
## Atmosphere & style
The presentation carries an authoritative, pragmatic, and slightly technical tone, blending legal advocacy with practical engineering solutions. Shulman presents Bright Data's stance with quiet confidence, drawing on sci-fi analogies like Blade Runner and principles from physics to illustrate the scaling challenges of AI agents. The vibe is that of a crisp, industry-focused keynote, where technical demonstrations and strategic announcements combine to offer a clear vision of the evolving 'Agentic Web'.
## Key takeaways
- [00:04](https://www.youtube.com/watch?v=7XsvnR8R2jE&t=4s) Bright Data advocates for open access to public web data, having won landmark federal lawsuits against Meta and X to establish the legal right to scrape public internet content.
- [00:29](https://www.youtube.com/watch?v=7XsvnR8R2jE&t=29s) AI agents act as proxies executing human intent, yet modern web anti-bot defenses incorrectly treat them as unauthorized bots and block their access at scale.
- [00:40](https://www.youtube.com/watch?v=7XsvnR8R2jE&t=40s) Web scraping friction for AI agents increases significantly when scaling from small internal testing environments to full production traffic.
- [06:13](https://www.youtube.com/watch?v=7XsvnR8R2jE&t=373s) AI agents require global proxy networks to route requests through specific country, city, or zip code IP addresses to obtain accurate local web content.
- [08:00](https://www.youtube.com/watch?v=7XsvnR8R2jE&t=480s) Circumventing website security blocks and CAPTCHAs requires dynamically managing request rates, latency, HTTP headers, and browser fingerprints.
- [08:55](https://www.youtube.com/watch?v=7XsvnR8R2jE&t=535s) Web Unlocker technology automatically handles retries, browser fingerprinting, and security hurdles to return clean HTML or Markdown from blocked sites.
- [09:50](https://www.youtube.com/watch?v=7XsvnR8R2jE&t=590s) Converting unrefined HTML into structured JSON or CSV data prevents excessive LLM token consumption and improves AI agent accuracy.
- [11:02](https://www.youtube.com/watch?v=7XsvnR8R2jE&t=662s) Bright Data integrates with the Model Context Protocol (MCP) to manage proxies, CAPTCHAs, and interactive page navigation directly within AI workflows.
- [12:05](https://www.youtube.com/watch?v=7XsvnR8R2jE&t=725s) A comparative demo showed standard web search tools getting blocked by Cloudflare, while the Bright Data MCP successfully bypassed defenses to extract target data.
- [14:09](https://www.youtube.com/watch?v=7XsvnR8R2jE&t=849s) Bright Data launched a free Web MCP Server offering up to 5,000 free monthly requests to help AI agents seamlessly navigate and extract web data.
## Detailed walkthrough
### [00:04](https://www.youtube.com/watch?v=7XsvnR8R2jE&t=4s) Bright Data's Role in the Legal Fight for Public Web Data
Ariel Shulman, Chief Product Officer at Bright Data, opens his presentation on a slightly philosophical note, referencing the high-profile media feud between Cloudflare and Perplexity regarding web scraping and bot protection. He frames Bright Data as standing directly in the center of this ecosystem as the global leader in public web data. Shulman shares that Bright Data has grown from a 20-person startup to a 400-employee company over his ten-year tenure, consistently advocating for open access to public web data—defined as any internet content accessible without a username or password.
To defend public web scraping rights, Bright Data has invested millions of dollars fighting high-stakes courtroom battles against major tech giants:
* **Meta Litigation:** Sued by Mark Zuckerberg's Meta, Bright Data won a landmark ruling in a California federal court. Shulman notes the irony that Facebook was actually a Bright Data customer using their network to scrape other sites, yet attempted to legally prevent others from scraping Meta's platforms.
* **X (Twitter) Litigation:** Sued by Elon Musk's X, Bright Data secured another legal victory affirming the right to collect publicly available web data.
Shulman emphasizes that technological innovation often outpaces the legal framework, making these legal precedents crucial for the entire AI and data ecosystem. Currently, Bright Data ingests over 2.5 petabytes of data daily into its archive, supplying millions of URLs across video, audio, and images that power major modern AI systems generating synthetic speech, music, and imagery.
---
[00:29](https://www.youtube.com/watch?v=7XsvnR8R2jE&t=29s) The Core Dilemma: Are AI Agents Humans or Bots?
Shulman shifts the discussion toward AI agents, asking the audience to consider whether AI agents should be classified as bots or as extension of humans. Drawing a parallel to the classic sci-fi film *Blade Runner*—where Harrison Ford's character hunts down replicants seeking extended lifespans—Shulman argues that AI agents act on behalf of human intent and execute explicit human instructions, effectively operating as human proxies.
Despite this human alignment, current anti-bot security systems on the open web overwhelmingly classify AI agents as unauthorized bots, blocking their automated requests at scale. Shulman demonstrates this vulnerability through a live recording of ChatGPT's recently introduced agentic research features (combining OpenAI's Deep Research and Operator agent capabilities). In the demonstration, Shulman asks the agent to find recommended headphones for bass guitar players:
1. The agent attempts to open **Amazon** to retrieve live pricing, but is instantly triggered by anti-bot measures and blocked.
2. The agent pivots to **Sweetwater** (a major online musical instrument retailer), where it is immediately blocked again.
```
+-------------------------------------------------------------------+
| HUMAN USER / AI AGENT |
| "Find bass guitar headphones on Amazon & Sweetwater" |
+-------------------------------------------------------------------+
|
v
+-------------------------------------------------------------------+
| TARGET WEBSITES |
| Amazon.com / Sweetwater.com / E-commerce Portals |
+-------------------------------------------------------------------+
|
[Trigger Anti-Bot / CAPTCHA Defenses]
v
+-------------------------------------------------------------------+
| BLOCKED REQUEST |
| Access Denied / Cloudflare Turnstile / 403 Forbidden |
+-------------------------------------------------------------------+
```
---
[00:40](https://www.youtube.com/watch?v=7XsvnR8R2jE&t=40s) Scaling Bottlenecks and Production Challenges
Drawing an analogy to quantum mechanics—where conventional physical laws break down at micro scales—Shulman explains that web scraping behavior behaves in reverse. During early development or internal MVP testing with small user bases, AI agents experience low access friction and rarely encounter blocks. However, the moment an agent scales to production traffic, anti-bot protections trigger aggressively, breaking functional workflows.
This structural barrier impacts both consumer-facing tools and enterprise AI agents. Even strictly internal enterprise agents regularly require public web access to retrieve contextual real-world data—such as a field technician's scheduling agent querying live weather updates from Google.
---
[06:13](https://www.youtube.com/watch?v=7XsvnR8R2jE&t=373s) Managing Geo-Locations and Circumventing CAPTCHAs
To ensure reliable, uninterrupted web access for AI agents, developers must address IP geolocation and anti-bot mitigation. Using the practical example of an AI agent designed to order pizza, Shulman points out that a query sent from New York versus Amsterdam must resolve to local pizzerias relative to the end user. If the agent makes requests from a mismatched cloud server IP address, target websites return irrelevant localized content or outright block the incoming connection.
Bright Data addresses this issue using its global proxy network:
* **100M+ IP Network:** Spans residential and commercial IP pools worldwide, granting control over routing down to specific countries, cities, or postal codes.
* **Granular Control:** Demonstrates sending API requests explicitly routed through specific jurisdictions (e.g., retrieving a Dutch IP address originating from Vodafone in North Holland).
```
+-------------------------------------------------------------------+
| AGENTIC REQUEST GENERATION |
| Local Query Execution (e.g., Target: Netherlands / Zip Code) |
+-------------------------------------------------------------------+
|
v
+-------------------------------------------------------------------+
| BRIGHT DATA PROXY NETWORK |
| 100M+ Residential & Commercial IPs (Country/City Targeting) |
+-------------------------------------------------------------------+
|
v
+-------------------------------------------------------------------+
| TARGET WEBSITE |
| Receives Localized, Human-Like Fingerprint & IP Address |
+-------------------------------------------------------------------+
```
In addition to geolocation, agents must actively bypass CAPTCHA challenges and security defenses. Modern website protections deploy dynamic fingerprinting and rate-limiting to curb malicious Denial of Service (DoS) attacks. Bright Data's lower-level infrastructure circumvents these triggers responsibly by measuring site latency, throttling requests, dynamically managing HTTP headers, and altering browser fingerprints to appear authentic.
---
[08:55](https://www.youtube.com/watch?v=7XsvnR8R2jE&t=535s) Overcoming Blocks with Web Unlockers and Web Scrapers
When a standard agent request encounters a traditional blocking screen or "dog page" on sites like Amazon, Bright Data utilizes its **Web Unlocker** architecture. The Web Unlocker accepts a target URL, executes background retries, manages browser fingerprints, and resolves security hurdles automatically, returning raw clean HTML or Markdown to the client.
To prevent agent token limits from being consumed by massive 1-megabyte HTML documents, raw data is processed through automated **Web Scrapers**:
* **Structured Output:** Accepts targeted product URLs and converts complex page data into clean, structured JSON or CSV formats containing structured data fields (up to 30+ columns per product).
* **Token Efficiency:** Minimizes token usage and context window consumption for downstream Large Language Models, significantly improving agent response accuracy and operational performance.
---
[11:02](https://www.youtube.com/watch?v=7XsvnR8R2jE&t=662s) Integrating Model Context Protocol (MCP) and Live Demo
Shulman explains how Model Context Protocol (MCP) serves as the standard abstraction layer connecting modern AI agents directly to Bright Data's collection infrastructure. Through the Bright Data MCP server, an agent's request triggers automated background execution: routing through optimal proxies, resolving CAPTCHAs, managing cookies, handling browser rendering, and executing interactive page steps (such as clicking specific size selectors on retail sites or typing text into input fields).
```
+-------------------------------------------------------------------+
| AI AGENT / LLM ENGINE |
+-------------------------------------------------------------------+
|
[Model Context Protocol]
v
+-------------------------------------------------------------------+
| BRIGHT DATA MCP SERVER |
| - Geo-Proxy Selection - Cookie & Header Management |
| - CAPTCHA Resolution - Headless Browser Rendering |
| - Page Interaction - HTML/Markdown/JSON Parsing |
+-------------------------------------------------------------------+
|
v
+-------------------------------------------------------------------+
| OPEN WEB SITES |
| Glassdoor, Amazon, Sweetwater, News, Social Platforms |
+-------------------------------------------------------------------+
```
Shulman conducts a side-by-side comparative live demonstration using two Claude Desktop instances attempting to scrape salary data for San Francisco software engineers from Glassdoor:
* **Standard Claude Desktop (Left):** Uses built-in native web search tools. It is blocked immediately upon encountering a Cloudflare Turnstile challenge.
* **Claude Desktop + Bright Data MCP (Right):** Takes slightly longer to execute because it actively bypasses site defenses, processes the target page, converts the extracted content into clean Markdown, and delivers complete salary data directly into Claude's prompt context.
---
[14:09](https://www.youtube.com/watch?v=7XsvnR8R2jE&t=849s) Announcing the Free Web MCP Server and Tooling Ecosystem
Shulman concludes by summarizing the architecture of the "Agentic Web"—where next-generation AI agents interact with an existing World Wide Web built primarily for human browser sessions. To bridge this gap, Bright Data acts as an uninterrupted middleware layer providing `/scrape`, `/search`, and `/crawl` API endpoints.
To support the developer community, Bright Data announces the launch of a free **Web MCP Server**:
* **Free Usage Tier:** Includes up to 5,000 free requests per month, with pay-as-you-go options for additional volume.
* **Built-in Capabilities:** Features native web searching, automated unlocking, full-site crawling, structured data extraction, and interactive browser actions (clicking and form filling).
* **Native Tool Integrations:** Deeply integrated directly within major developer platforms and agent frameworks.
* **Exhibition Booth:** Invites attendees to join their official launch event at Booth #2011 in the Ai4 exhibition hall.
## Notable quotes
> [00:00](https://www.youtube.com/watch?v=7XsvnR8R2jE&t=0s) Your AI agents are only as intelligent as their data allows them to be. — Ariel Shulman
> [00:29](https://www.youtube.com/watch?v=7XsvnR8R2jE&t=29s) are AI agents humans or bots? — Ariel Shulman
## Mentioned
Ariel Shulman, Bright Data, Cloudflare, Perplexity, Meta, Mark Zuckerberg, X, Twitter, Elon Musk, Blade Runner, Harrison Ford, ChatGPT, OpenAI, Deep Research, Operator, Amazon, Sweetwater, Google, Vodafone, Model Context Protocol, Claude Desktop, Glassdoor, Ai4
## Video description
Ariel Shulman, Chief Product Officer, Bright Data
Abstracts
Your AI agents are only as intelligent as their data allows them to be. Explore how innovative organizations equip autonomous AI agents with real-time web capabilities, transcending traditional search engine limitations. In this session, you'll discover how to integrate advanced search techniques, intelligent crawling, sophisticated navigation, LLMs, social media, news and web archives. Learn to build AI agents that deliver accurate, timely, and contextually nuanced results - all while upholding rigorous ethical standards.
胜诉Meta后，Bright Data发布免费AI Agent代理工具
反爬虫防御系统在大规模场景下会系统性拦截自主AI Agent，需要专用代理与解锁中间件以维持实时公开网络访问。
Ariel Shulman探讨了AI Agent在访问公开网页数据时面临的法律与技术障碍，指出Agent实质上是人类意图的代理执行者，却普遍遭到反爬虫保护机制的拦截。他概述了Bright Data如何通过与Meta和X等科技巨头进行具有里程碑意义的法律诉讼，以捍卫公开网络数据抓取的合法权利。演讲详细介绍了Bright Data的基础设施，包括其拥有超1亿IP的代理网络、Web Unlocker、动态验证码（CAPTCHA）绕过技术，以及将原始HTML转换为节省Token的结构化数据的自动化网页抓取工具。最后，Shulman使用Claude Desktop与Bright Data的模型上下文协议（MCP）服务器进行了实时对比演示，并宣布推出免费层级，方便开发者将Agent直接接入实时网络能力。
Bright Data 在与 Meta 和 X 的关键联邦诉讼中接连胜诉，确立了抓取公开网络数据的合法权利，目前每日向数据归档摄取超过 2.5 PB 数据以支持前沿 AI 系统。 (0:04)
AI Agent 本质上是人类意图的执行代理，但开放网络的反爬虫与安全验证系统普遍将其判定为未授权 Bot，导致其在进入生产流量规模后频繁遭遇拦截。 (0:29)
依托拥有逾 1 亿节点的全球代理网络与 Web Unlocker 技术，Bright Data 能够精确模拟本地请求并自动化解验证码阻拦，同时将网页转换为结构化数据以节省 LLM Token 消耗。 (8:55)
Bright Data 宣布推出面向开发者的 Web MCP Server 并提供每月 5,000 次免费请求，可直接通过 Model Context Protocol 为 AI Agent 赋予网页搜索、解锁、抓取及交互能力。 (14:09)
Reading summary
Ariel Shulman, Chief Product Officer at Bright Data, argues that while AI agents operate as direct extensions of human intent, modern web infrastructure predominantly misclassifies and blocks them as unauthorized bots. To prevent automated agent workflows from breaking at production scale, Shulman advocates for legal protections around public data access and the deployment of proxy and protocol middleware.
Legal Battles Over Public Web Access (0:04)
Bright Data defines public web data as any online material accessible without requiring a username or password. Over his decade-long tenure as the company expanded from a 20-person startup to 400 employees, Shulman notes that Bright Data has committed millions of dollars to litigate and secure legal precedents for open web collection in US federal courts. Notably, the firm won a California federal lawsuit against Meta—a company Shulman points out was simultaneously a Bright Data customer utilizing scraping networks elsewhere—as well as a separate lawsuit against Elon Musk's X (formerly Twitter).
Shulman emphasizes that technological progress routinely moves faster than judicial frameworks, making affirmative court rulings critical for the broader artificial intelligence industry. Bright Data presently ingests over 2.5 petabytes of public data per day, curating multi-modal web inputs across images, audio, and video to supply the training pipelines of major synthetic generative models.
The Production Bottleneck Facing AI Agents (0:29)
Shulman presents a fundamental classification dilemma: whether autonomous software agents should be treated as unauthorized bots or as legitimate human proxies executing human intent. While agents serve human objectives, prevailing security tools treat automated traffic identically to malicious actors. Drawing a comparison to quantum mechanics—where behavior alters depending on scale—Shulman notes that early-stage agent MVPs and small-scale prototypes typically browse the web without obstruction, but immediately break down under anti-bot triggers once deployed to production traffic volumes.
To demonstrate this limitation, Shulman reviewed a live test using ChatGPT's research and Operator features tasked with finding bass guitar headphones. When querying Amazon for live prices, the agent was immediately intercepted by anti-bot defenses; when pivoting to Sweetwater, it was halted again by Cloudflare Turnstile verification. Shulman stresses that this friction disrupts both external consumer tools and internal enterprise agents that rely on routine real-world data, such as field technician schedulers fetching local weather.
Resolving Geolocation, Fingerprints, and Token Bloat (6:13)
Functional agent interaction requires navigating IP geolocations, browser fingerprints, and token limits. For localized tasks—such as an agent ordering food—requests originating from mismatched data-center IP addresses result in localized errors or outright connection drops. Bright Data counters this with a network of over 100 million residential and commercial IPs, allowing granular routing down to specific countries, municipalities, or postal codes.
Beyond IP targeting, the company's Web Unlocker dynamically adjusts request throttling, alters HTTP headers, and simulates genuine browser fingerprints to navigate security challenges without triggering Denial of Service safeguards. To address token consumption when passing uncompressed web pages to large language models, automated Web Scrapers parse 1-megabyte HTML files into structured JSON or CSV schemas containing over 30 standardized fields, preserving LLM context windows and improving inference precision.
Connecting Agents via Model Context Protocol (11:02)
To standardize how agents interface with data collection pipelines, Bright Data adopted Anthropic's Model Context Protocol (MCP). The Bright Data MCP server functions as an abstraction layer between the LLM and the open web, managing headless browser rendering, proxy assignment, cookie states, CAPTCHA bypasses, and physical web interactions such as button clicking or form filling.
During a live comparative evaluation using two Claude Desktop clients querying San Francisco software engineer salaries on Glassdoor, standard Claude native search was halted by Cloudflare security. In contrast, Claude equipped with the Bright Data MCP server successfully navigated the security checks, rendered the page, and ingested the extracted salary figures into Claude's prompt context as clean Markdown. Shulman accompanied the demonstration with the release of a free-tier Web MCP Server offering developers 5,000 free monthly requests across search, scraping, and crawling endpoints.
Shulman asserts that the emerging 'Agentic Web' requires dedicated middleware because modern websites were engineered exclusively for human browser sessions. While MCP and automated proxy pipelines offer a bridge for real-time information retrieval, the ecosystem remains constrained by an ongoing technical arms race between website anti-bot protections and automated agent access.
[0:00] Ariel Shulman: Your AI agents are only as intelligent as their data allows them to be.
[0:29] Ariel Shulman: are AI agents humans or bots?
Original transcript
[0:04] Okay, I'm going to keep it maybe a bit
[0:05] more high level. Some would say even
[0:07] philosophical uh with this presentation.
[0:08] By show of hands, how many people have
[0:10] heard about the perplexity/cloudfare
[0:13] flight fight that happened on the press
[0:15] during the press year? Okay, quite a
[0:17] few. So maybe you know or you haven't,
[0:20] but Cloudflare protects websites from
[0:23] malicious bots. Perplexity is getting
[0:25] information. There was a lot of back and
[0:26] forth in the press and basically we are
[0:28] what we are in bright doing. Bright data
[0:30] is kind of in the middle of all of this.
[0:32] So that's what we're going to talk
[0:33] about. So Bright Data is the global
[0:36] leader in in public web data. We are
[0:38] basically the the company the the
[0:41] world's largest web scraping company and
[0:44] I've been with the company for over 10
[0:45] years. See it was amazing to see this
[0:47] grow from 20 to about 400 people. This
[0:49] is not a startup anymore and um during
[0:52] the process we have been kind of leading
[0:53] the fight in the you know for free
[0:56] access to public web data. Public web
[0:58] data means everything that's available
[0:59] on the internet without any username or
[1:01] password to anyone. Okay. So as part of
[1:04] that we were sued
[1:06] a few years ago by Mr. Mark Zuckerberg
[1:09] back when he had a different hairstyle
[1:12] and we won. It's interesting because
[1:15] Facebook was actually a customer of
[1:16] bright data scraping other people but
[1:18] they also thought that we were not
[1:20] allowed to scrape their websites. So f
[1:22] federal judge in California ruled in our
[1:24] favor which was very important and Mr.
[1:27] Musk sued us as well and again we won
[1:31] against Twitter. Twitter again did not
[1:33] think that we could scrape public web
[1:36] data. I stress public and we want and we
[1:39] invested millions of dollars millions of
[1:41] dollars in these lawsuits and I think
[1:42] this is very important because some
[1:45] technology is kind of ahead of the law
[1:46] of the law and this is a good case of
[1:48] that. So all of these victors are really
[1:50] kind of a a very important precedent to
[1:52] what's going on in the public web data
[1:54] world.
[1:56] So in general we as a web scraping
[1:58] company collect all tons and tons and
[2:00] tons of public web data. Public web data
[2:02] again could be prices, reviews, etc. Uh
[2:05] so we're talking about huge amounts 2.5
[2:08] pabytes a day uh that are keep being
[2:10] added to this kind of archive and all of
[2:13] these millions of URLs for audio and
[2:15] video and images are used by all the
[2:18] companies that you know to generate
[2:20] images to generate songs to generate
[2:22] human voices and things like that.
[2:25] That's kind of where AI gets its gets
[2:27] its data.
[2:29] But what I'd like to talk about today is
[2:31] actually maybe a bit more philosophical
[2:34] and that is AI agents. So are AI agents
[2:38] bots or humans? By show of hands, who
[2:41] has seen Bladeunner here?
[2:44] Great. It's amazing movie. And by the
[2:47] way, even the sequel is very, very good.
[2:50] And in Bladeunner, Harrison Ford is
[2:52] tasked with the task of killing bots
[2:56] that want to extend their life. So, our
[2:59] AI agents, bots are humans. I submit
[3:03] that in a way they are humans because
[3:05] they are in fact carrying out our
[3:08] desires. They're doing what we tell
[3:10] them. Um, but, and this is really
[3:14] interesting,
[3:15] the web doesn't seem to think that AI
[3:17] agents are humans.
[3:19] They get blocked. Okay? They get blocked
[3:21] a lot. And that's what we're going to
[3:23] talk about. I will show you how they get
[3:24] blocked. I will show you how you can get
[3:25] around blocking. we can see how that
[3:28] makes sense. So let's look at this. This
[3:30] is chat GPT the chat GPT agent which was
[3:33] just announced last month. It's a
[3:35] combination of deep research and the
[3:37] operator if you remember and I threw a
[3:40] query at at chat GPT. I I like to play
[3:42] bass. I needed headphones. So I said
[3:44] please give me research, you know, find
[3:46] me some good headphones for bass
[3:48] players. And what you're going to see
[3:49] now is not a mockup. It's a recording
[3:52] that I just grabbed on my Mac. And this
[3:54] is how it went. So it started looking
[3:57] okay and then it said I'm going to go to
[3:59] Amazon okay opening Amazon for a price
[4:02] and oop it gets blocked
[4:05] okay
[4:06] >> that does happen a lot
[4:07] >> it does happen a lot so this is and this
[4:09] is not an iso isolated coincidence if I
[4:12] go it kept trying it went to another
[4:14] website called sweetwater this is kind
[4:16] of the Amazon of of musical instruments
[4:19] so it's opening sweet of water for
[4:20] results and surprise surprise boop it's
[4:24] blocked
[4:25] Okay, so these things happen. They
[4:29] happen to everyone and they happen at
[4:33] scale and this is the tricky part. Okay,
[4:37] these things show up in production. AI
[4:40] agents these days, they are very low on
[4:42] the adoption curve. So people are
[4:43] developing AI agents, but in terms of
[4:45] like the real massive usage hasn't
[4:47] really started yet. Okay. And if you're
[4:50] familiar with quantum mechanics, in
[4:52] quantum mechanics, when things become
[4:53] really really small, all the laws of
[4:55] physics kind of break down. Everything
[4:57] goes crazy. And uh my point is that with
[5:00] with web access, it's exactly the
[5:02] opposite. So when you are doing your
[5:04] MVP, you're in the office, you're doing
[5:05] some things, you have you know some of
[5:07] your colleagues testing the system, it's
[5:09] fine. That's small scale. As soon as you
[5:11] go c above a certain threshold, things
[5:14] go completely crazy. And that's where
[5:18] you know the the the whole notion of the
[5:21] traditional web scraping which is what
[5:23] we've been doing for the last 10 years
[5:25] becomes super relevant for AI agents
[5:26] because it g gives them access to these
[5:29] websites and again it doesn't matter if
[5:31] you're building enterprise AI agents or
[5:34] like public AI agents because even
[5:35] enterprise AI agents very often need
[5:38] some access to the outside world. Let's
[5:40] say that someone builds an AI agent for
[5:42] I don't know a technician that needs to
[5:44] schedule their visits tomorrow. Maybe
[5:46] the technician is going to ask you know
[5:47] what will be the weather. They want to
[5:48] know if they need to bring an umbrella
[5:50] or not. That will go to Google and try
[5:51] to get the weather responses. Okay, this
[5:53] kind of external web access gets blocked
[5:56] at a very small scale but it's very
[5:58] hard. Uh so if there's one thing I'd
[6:01] like you to remember is that these
[6:02] things, you know, creep up on you. It
[6:04] just happens and then you don't know
[6:06] what happens. It's very difficult. Um,
[6:09] so this is that's the challenge. It
[6:11] really shows up in production.
[6:13] So how can you make the AI agent work
[6:15] work reliably? How can you make this
[6:18] happen? Few things and I'm not going to
[6:21] get into too many details here, but
[6:22] let's talk about the things that makes
[6:24] make most sense. Let's say that you have
[6:26] an AI agent that is all about ordering
[6:28] the best pizza. Okay, you can use
[6:30] natural language to order pizza just as
[6:32] an example. Clearly, if your users are
[6:35] in New York or in Amsterdam, you need to
[6:37] give them a completely different answer.
[6:40] Okay? There's no doesn't make sense to
[6:42] recommend to someone in New York a pizza
[6:44] joint in Amsterdam or vice versa. So,
[6:46] the IP address that your AI agent uses
[6:49] to access the website has real
[6:51] implications on how the the website will
[6:53] come back to you with an answer. So, you
[6:55] need to control your location.
[6:57] So, in bright data, this is kind of our
[6:59] bread and butter from many many years
[7:01] ago. We have a huge proxy network over
[7:04] 100 million IPs all around the world.
[7:06] And what you can do is you can very
[7:08] easily kind of tell your agent to use a
[7:11] specific location. You can do that even
[7:13] on a on a per country level, city, zip,
[7:17] very very detailed. So what you see here
[7:19] in front of you is a request. Okay? And
[7:22] basically it's saying please give me an
[7:24] an address sorry a proxy in Netherlands.
[7:28] And you can see that the response out
[7:29] comes back. Yes, this was country
[7:31] Netherlands and it's in North Holland
[7:33] with Vodafone or whatever. So you need
[7:36] to be able to access where you are going
[7:38] from because if you are using the
[7:40] incorrect IP address, you will get
[7:41] either a strange result or blocked. Some
[7:43] websites need to get accessed from the
[7:46] same country. Okay, so that is the
[7:48] concept of the IP address. But then even
[7:52] if you have the IP address or even if
[7:54] you're in the correct location, quite
[7:55] often your agent will get blocked. And
[7:58] the idea here is to try to avoid the
[8:00] blocking. So avoid the capture. Okay.
[8:03] And this is as it says here getting much
[8:05] much harder. You can try to do that by
[8:08] using the proxies what we saw by using
[8:10] multiple types of proxies by customizing
[8:13] headers and fingerprints. And basically
[8:15] the idea is to make the website
[8:17] understand that you are a human. And I
[8:20] want to reiterate here, this is very
[8:22] important. Websites have protections.
[8:25] Protections against malicious
[8:26] activities. For example, DOS attacks
[8:28] where people go and hit the website a
[8:29] thousand requests a minute. This is
[8:31] different. This is when you go
[8:33] specifically to get some specific bits
[8:35] of information and it's done in our you
[8:38] know lower level of the stack in a very
[8:40] responsible way that measures the
[8:41] response time of the website and
[8:43] throttles back and does all sorts of
[8:45] things to make sure that sure that we do
[8:47] the right thing. So, it's getting
[8:48] harder. As it says here, the best way to
[8:50] solve the capture is to avoid it.
[8:52] Captures can be sold if they appear. So,
[8:55] let's see what this looks like. Let's
[8:57] see. Let's say that you want to go to
[8:59] Amazon and retrieve a page, a product
[9:01] page. You have an AI agent that does
[9:03] shopping, okay? Price comparison. So,
[9:05] you send a request. Typically, this this
[9:08] is what you're going to see, okay? It
[9:10] will be this or some other kind of dog
[9:12] or something that's going to tell you
[9:14] that you have not you have been blocked.
[9:16] If you send the same URL that you see
[9:18] here through what we call the web
[9:20] unlocker which is the culmination of all
[9:23] the technology stack that we have with
[9:25] proxies and you know fingerprints and
[9:27] all ton of ton of things then you can
[9:30] see that you're sending the same URL
[9:32] into this unlocker thing and it will
[9:34] come back it will come back under the
[9:36] hood there will be tons of things
[9:37] happening okay you don't need to know
[9:38] that but there there'll be tons of
[9:40] things happening and retries and all
[9:41] sorts of things that will eventually
[9:43] just get you the data so input you URL
[9:46] output HTML or markdown but you will get
[9:49] the data a guaranteed.
[9:51] So this is what we call the web
[9:52] unlocker.
[9:54] Now this kind of web data can also be
[9:57] and this is especially important for web
[9:59] agents parsed or structured because you
[10:01] don't want to get a huge like one
[10:03] megabyte HTML page to your AI agent.
[10:06] It's not very useful. So it is possible
[10:08] to use what we call web scrapers which
[10:10] we have a you know huge library of. And
[10:12] you can see here again without getting
[10:14] getting into details a set of URLs.
[10:16] Okay, these are three Amazon products.
[10:19] You send that to the API endpoint and
[10:21] what you get back is in this case a CSV.
[10:25] So a really nice structured table. This
[10:27] has like 30 columns basically every
[10:30] field that you saw on the page nicely
[10:32] filled out and we take the the
[10:35] information from the page put it in and
[10:36] make that for you in a nice CSV or JSON
[10:38] file. And what this is, you know,
[10:41] crucial for when you are doing AI agents
[10:43] is you are getting much better
[10:45] performance. You you're you get a small
[10:47] response. It eats up the minimum amount
[10:49] a minimum amount of tokens. And the fact
[10:52] that the data is already structured
[10:53] means that you will get better results
[10:55] from your AI agent.
[10:57] So this is the uh the parsing part.
[11:02] Since we're talking about AI agent, MCP
[11:05] is like the magic word. Everyone I think
[11:06] probably this word this acronym has been
[11:08] thrown around like a gazillion times in
[11:10] this conference already. So MCP is is
[11:12] the way to do this. All of our solutions
[11:14] can be accessed directly but MCP is the
[11:16] way to do this. And I'd like to walk you
[11:17] through how an AI agent connects to a
[11:20] website and back. So via the right MCP
[11:24] you send a request. What will happen
[11:27] under the hood is that we will choose
[11:30] the best jail or do or or choose what
[11:32] you have actually asked us to use. solve
[11:35] the captures if any, set all sorts of
[11:38] cookies and headers to uh to unlock the
[11:41] website. If there is some kind of
[11:42] blocking, we will do under the hood IP
[11:44] retrying, all sorts of things to
[11:46] eventually get you um the data.
[11:49] In many cases, websites need to be
[11:53] rendered. So that means that you open a
[11:55] page, there's need there are a few
[11:57] clicks that need to be that take place
[11:58] on the page in order for the data to be
[12:00] available. For example, you want to go
[12:02] and find the dress in a different sites.
[12:04] So the one page that have the dress and
[12:05] there's like buttons where you can
[12:07] choose the size. So you that needs to be
[12:09] clicked. We know how to do that too.
[12:12] Sometimes there's text that needs to be
[12:14] to be inputed into text boxes. So we
[12:16] know how to do that as well. Eventually
[12:19] you get your information from the
[12:20] website and if needed it will be parsed
[12:23] and structured so you will so that it
[12:24] will be minimal in terms of tokens and
[12:26] very effective for for your for needs.
[12:31] So what I'd like to show you is
[12:33] basically a demo. I'm going to pause
[12:34] this for a second.
[12:37] And here on the left we see uh these two
[12:40] two instances of cloud desktop. On the
[12:43] left is just direct access, okay, with
[12:46] the cloud built-in web search
[12:48] capabilities. And on the right it's
[12:49] connected or chained to the bright MCP
[12:51] server. What am I asking? I'm going to
[12:54] need my glasses for this.
[12:59] So use your web tool and get the
[13:01] contents of this page. It's a glass door
[13:03] page. Okay. About salaries of engineers
[13:05] in San Franc in San Francisco. So we're
[13:08] going to do click on both at the same
[13:10] time. Oops. Sorry. That was not supposed
[13:12] to happen.
[13:15] Click on both at the same time. And
[13:17] let's see what what's happening on the
[13:19] left and on the right.
[13:21] Here goes.
[13:24] So on the left
[13:27] cloud desktop is going to say I've tried
[13:29] but I've been blocked. And what happens
[13:31] under the hood is that it was it got a
[13:33] cloudflare turn style. This is what it
[13:35] looks like under the hood. On the right
[13:37] hand side you can see that this is
[13:39] taking a bit longer because there is
[13:40] work to be done here. But the bright mcp
[13:44] goes through all the steps that I showed
[13:45] you earlier to get the HTML of the page
[13:48] provide in this case markdown for the
[13:50] cloud desktop and cloud desktop is able
[13:52] to generate a response. So this is just
[13:55] a simple example but uh these things
[13:59] happen all the time and if they have not
[14:00] happened to you yet they will and
[14:02] probably at the worst possible time. So
[14:04] just be aware.
[14:09] So kind of to uh to summarize
[14:12] I would say that the uh agentic web is
[14:15] if you look at the top layer there is
[14:17] kind of the new internet okay and the
[14:19] new internet which is aentic web tries
[14:22] to communicate with the old worldwide
[14:23] web which was made for users and in many
[14:26] cases you get blocked. The way we see
[14:28] this in bright data is that when you add
[14:31] the bride data
[14:33] you know stack along the way we either
[14:36] do these are the different endpoints
[14:38] that we have slc scrape search crawl
[14:40] etc. We will allow this uninterrupted
[14:42] access to the agent so that you it will
[14:46] be able to accomplish its tasks.
[14:50] So wrapping up uh we got you back. We
[14:53] are actually announcing today a free web
[14:57] MCP server which anyone can use. Uh it
[15:01] will automatically give you ability to
[15:04] search the web using different search
[15:06] engines. Unlock any web page using the
[15:08] unlocker that you saw here. Crawl
[15:10] websites. Get structured data if you
[15:12] need to do that and not just markdown or
[15:14] HTML. Interact with websites. So you can
[15:16] actually ask claude to open a website
[15:19] start clicking around. It will do that.
[15:21] And this is deeply integrated with many
[15:23] tools that you see here. I'm not talking
[15:25] about like basic integration. You can
[15:26] actually find this in the tools
[15:28] themselves. If you go if you search for
[15:29] B data inside these tools, you're going
[15:32] to see this. It's up to 5,000 requests a
[15:34] month. It's enough to play around. If
[15:35] you exceed that, it's very easy to to
[15:37] just, you know, add some credit and go
[15:39] with the pay as you go. So, we have a
[15:41] launch party for this. Uh we are going
[15:43] to go and and do this now. And you're
[15:46] more than welcome to join us in the
[15:47] booth. We are in 2011. as you enter the
[15:50] exhibition hall right in front of you.
[15:53] That's it. Thank you very much everyone.
