Skip to content

JOBUZO

  • News
  • Indonesia
  • Toggle search form
Cloudflare launches a marketplace that lets websites charge AI bots for scraping

Cloudflare launches a marketplace that lets websites charge AI bots for scraping

Posted on 1 July 2025 By jobuzo

Cloudflare, a cloud infrastructure provider that serves 20% of the web, announced Tuesday the launch of a new marketplace that reimagines the relationship between website owners and AI companies — ideally giving publishers greater control over their content.

For the last year, Cloudflare has launched tools for publishers to address the rampant rise of AI crawlers, including a one-click solution to block all AI bots, as well as a dashboard to view how AI crawlers are visiting their site. In a 2024 interview, Cloudflare CEO Matthew Prince told TechCrunch these products were laying a foundation for a new type of marketplace in which publishers could distribute their content to AI companies and be compensated for it.

Now, Cloudflare is bringing that marketplace to life.

It’s called Pay per Crawl, and Cloudflare is launching the “experiment” in private beta on Tuesday. Website owners in the experiment can choose to let AI crawlers, on an individual basis, scrape their site at a set rate — a micropayment for every single “crawl.” Alternatively, website owners can choose to let AI crawlers scrape their site for free, or block them altogether. Cloudflare claims its tools will let website owners see whether crawlers are scraping their site for AI training data, to appear in AI search responses, or for other purposes.

Here’s what website owners see in Pay per Crawl (Credit: Cloudflare)

At scale, Cloudflare’s marketplace is a big idea that could offer publishers a potential business model for the AI era — and it also places Cloudflare at the center of it all. The launch of the marketplace comes at a time when news publishers are facing existential questions about how to reach readers, as Google Search traffic fades away and AI chatbots rise in popularity.

There’s not a clear answer for how news publishers will survive in the AI era. Some, such as the New York Times, have filed lawsuits against tech companies for training their AI models on news articles without permission. Meanwhile, other publishers have struck multi-year deals to license their content for AI model training and to have their content appear in AI chatbot responses.

News :<div>12 weeks' jail for school IT support technician who took upskirt videos of teachers</div>

Even so, only large publishers have struck AI licensing deals, and it’s still unclear whether they provide meaningful sources of revenue. Cloudflare aims to create a more durable system where publishers can set prices on their own terms.

The company also announced Tuesday that new websites set up with Cloudflare will now, by default, block all AI crawlers. Site owners will have to grant certain AI crawlers permission to access their site — a change Cloudflare says will give every new domain “the default of control.”

Several large publishers, including Conde Nast, TIME, The Associated Press, The Atlantic, ADWEEK, and Fortune, have signed on with Cloudflare to block AI crawlers by default in support of the company’s broader goal of a “permission-based approach to crawling.”

The business model that many of these publishers relied on for decades is slowly becoming unreliable. Historically, online publishers have allowed Google to scrape their sites in return for referrals in Google Search, which translated to traffic to their sites, and ultimately, ad revenue.

However, new data from Cloudflare suggests that publishers may be getting a worse deal in the AI era than in the Google Search era. While some websites cite ChatGPT as a major traffic source, that doesn’t appear to be the case broadly.

This June, Cloudflare says it found that Google’s crawler scraped its websites 14 times for every referral it gave them. Meanwhile, OpenAI’s crawler scraped websites 17,000 times for every one referral, while Anthropic scraped websites 73,000 times for every referral.

News :Migrant acquitted in first trial over US border military zones

Meanwhile, OpenAI and Google are building AI agents that are designed to visit websites on behalf of users, collect information, and deliver it back to users directly. A future in which these tools are mainstream has huge implications for publishers that rely on readers visiting their sites.

Cloudflare notes that the “true potential” of Pay per Crawl may emerge in an “agentic” future.

“What if an agentic paywall could operate at the network edge, entirely programmatically? Imagine asking your favorite deep research program to help you synthesize the latest cancer research or a legal brief, or just help you find the best restaurant in Soho — and then giving that agent a budget to spend to acquire the best and most relevant content,” Cloudflare said in a blog post.

To participate in Cloudflare’s experimental marketplace, AI companies and publishers must both be set up with Cloudflare accounts. In their accounts, both parties can set rates at which they’d like to buy and sell a “crawl” of the publisher’s content. Cloudflare acts as the intermediary in these transactions, charging the AI company and distributing the earnings to the publisher.

Cloudflare spokesperson Ripley Park tells TechCrunch there are no stablecoins or cryptocurrency involved in Pay per Crawl at this time, even though many have suggested digital currency would be perfect for something like this.

Cloudflare’s marketplace feels like a bold vision for the future that requires a lot of publishers and AI companies to get on board. Still, there’s no guarantee publishers will get a good deal, and convincing AI firms to participate could be tough, given they’re currently scraping content for free.

Nevertheless, Cloudflare seems like one of the few companies in a position to make a marketplace like this happen.

Cloudflare launches a marketplace that lets websites charge AI bots for scraping


News

Post navigation

Previous Post: China urges immediate ceasefire, humanitarian access in Gaza
Next Post: El Chapo drug cartel reportedly tracked and killed informants by hacking an FBI phone

Related Posts

Senate OKs Trump bill, House showdown looms Senate OKs Trump bill, House showdown looms News
Akazawa talks of 'dense fog' in U.S. trade talks ahead of G7 summit Akazawa talks of ‘dense fog’ in U.S. trade talks ahead of G7 summit News
FACT CHECK: Trump claims LA protesters came in with bricks, bore foreign flags FACT CHECK: Trump claims LA protesters came in with bricks, bore foreign flags News

Latest

  • Senate OKs Trump bill, House showdown looms
  • Chinese firms eye Hong Kong listings after mainland investors’ buying spree
  • Chinese opera performance brings cultural exchange to central Vietnam
  • Concerns over US’s declining stockpiles lead to halt in weapons shipments to Ukraine
  • Russia says certain forces are out to wreck ties between it and Azerbaijan, RIA reports
  • Australia to tighten childcare laws after Melbourne worker charged with child sex offences
  • African Union helicopter crashes in Mogadishu airport in Somalia
  • Trump says Israel has agreed to terms for 60-day Gaza ceasefire
  • Japan, US, Australia, India to bolster critical mineral cooperation
  • Australian airline Qantas hit by massive cyberattack

Copyright © 2025 JOBUZO. Disclaimers | Privacy Policies

Powered by PressBook Masonry Blogs