Cloudflare launches a marketplace that lets websites charge AI bots for scraping

Cloudflare, a cloud infrastructure provider that serves 20% of the web, announced on Tuesday the introduction of a new marketplace that reinterpreted the relationship between the website owner and AI company.

Last year, Cloudflare Tools started publishers to the rampant climb of AI -Crawler, including A One-Click solution to block all AI botsas well as a Dashboard to see how Ki -Crawler visit your website. In an interview from 2024, Matthew Prince, CEO of Cloudflare, shared techcrunch that these products would lay a basis for a new type of market on which publisher could distribute their content to AI companies and compensate for it.

Now Cloudflare awakens this marketplace to life.

It is called Pay Pro Crawl and Cloudflare, the “experiment” starts in private beta on Tuesday. Website owner of the experiment can decide to scratch your website on an individual basis at an individual basis at a fixed speed -a micropayment for each “crawl”. Alternatively, website owners can have the KI -Crawler -scratch or block the overall. Cloudflare claims that his tools can see the website owner whether Crawler can scrape off your website for AI training data, in AI search answers or for other purposes.

On the scale, Cloudflare marketplace is a big idea that could offer the publishers a potential business model for the KI era – and also focuses on Cloudflare. The marketplace starts at a time when news publishers are confronted with existential questions to achieve readers, while the Google Search traffic faded and the AI ​​chatbots increase in popularity.

There is no clear answer to how news publishers will survive in the AI ​​era. Some, like the New York Times, submitted Complaints against technology companies for the training of your AI models in news articles without permission. In the meantime, other publishers have achieved several years to licens their content For AI model training and the appearance of your content in AI chat bot.

Nevertheless, only large publishers have concluded AI license contracts, and it is still unclear whether they provide meaningful sources of income. Cloudflare aims to create a more durable system in which publishers can set prices on their own conditions.

The company also announced on Tuesday that new websites with Cloudflare will now block all AI crawlers by default. Site owners must give certain Ki -Crawler’s permission to access their website. A change Cloudflare will give every new domain “the standard of control”.

Several large publishers, including Conde Nast, Time, The Associated Press, The Atlantic, Adweek and Fortune, have registered with Cloudflare in order to block the AI ​​crawlers by default in order to support the wider goal of the company of a “permission to crawl”.

The business model, to which many of this publishers have risen for decades, is slowly becoming unreliable. In the past, online publishers have allowed Google to scrape their websites for transfers in Google Search, which were translated on traffic on their websites and ultimately advertising revenues.

However, new data from Cloudflare indicate that publishers may depend on a worse offer in the AI ​​era than in the Google Search -era. While Some websites indicate Chatgpt as an important source of trafficThat doesn’t seem to be the case in general.

In this June Cloudflare found that Google’s Crawler had scratched his websites 14 times for each transfer he had given. In the meantime, Openai’s Crawler scraped the websites 17,000 times for each transfer, while anthropic websites scrape 73,000 times for each transfer.

In the meantime, Openaai and Google KI agents are building them down Visit websites on behalf of the userCollect information and return it directly to users. A future in which these tools are mainstream has a big impact on publishers that rely on readers who visit their websites.

Cloudflare notes that the “true potential” of the payment per crawl can occur in a “acting” future.

“What if an Agent -Paywall could operate completely program -controlled on the Network EDGE? Imagine you ask your preferred Deep research program to help you synthesize the latest cancer research or a legal letter, or help you find the best restaurant in Soh -and then give the agent a budget to acquire the best and relevant content,” said, “said Cloudflare in a blog post.

In order to take part in Cloudflare in the experimental marketplace, AI companies have to be set up with Cloudflare accounts. In their accounts, both parties can set tariffs to which they want to buy and sell a “crawl” of the publisher’s content. Cloudflare acts as an intermediary in these transactions, calculates the AI ​​company and distributes the result to the publisher.

The Cloudflare spokesman Ripley Park announces Techcrunch that at that time it does not involve stable coins or cryptocurrency with the payment per crawl, although many have proposed Digital currency would be perfect for something like that.

Cloudflare’s marketplace feels like a brave vision for the future, which requires many publishers and AI companies to go on board. Nevertheless, there is no guarantee publishers that receive good business, and it could be difficult to convince the AI ​​companies, as they are currently scraping up for free.

Nevertheless, Cloudflare seems to be able to enable a marketplace to enable a marketplace.