Big if True: Cloudflare is taking steps to combat unregulated data scraping carried out by AI crawlers. However, the company's CEO says meaningful progress will also require cooperation from major AI companies, and they might end up getting none.
Cloudflare recently announced a new "pay-per-crawl" system aimed at pushing back against AI companies that continue to scrape the open web without paying a cent. According to CEO Matthew Prince, the effort should be supported by the very companies doing the crawling.
Prince joined a discussion on X to elaborate on Cloudflare's new initiative. He confirmed that Google's Gemini bot is blocked by default and stated that the company is working to pressure Google into offering a practical method for blocking AI-specific features – like Answer Box and AI Overviews – while still allowing traditional search indexing bots.
Prince emphasized that it is indeed possible to block Google's AI crawlers without disrupting its standard web indexing processes. He added that Cloudflare is developing a solution to make this separation easier for site owners. While traditional content indexing has long been the backbone of the web, the rise of the so-called Zero Click Internet is threatening to undermine that balance.
Big Tech and AI chatbot ventures are aggressively exploiting the public internet with a relentless swarm of bots designed to harvest every piece of human-created content available online. Even robust infrastructure projects like Wikipedia, engineered to handle billions of global users, are now struggling under the unregulated strain of AI-driven scraping.
According to Prince, the company's new plan aims to preserve the web's accessibility for human users while curbing AI-related abuse. But what if the strategy fails, and Google simply refuses to cooperate with the content delivery network?
In a worst-case scenario, Prince said, it may become necessary to introduce legal mandates requiring companies like Google to separate their AI crawlers from standard search indexing bots. He noted that doing so wouldn't be especially difficult, though he remains hopeful that new anti-bot legislation can be avoided.
As technology evolves at breakneck speed and political influence wanes in comparison to Big Tech lobbying power, enforcing reasonable regulations against digital overreach is becoming increasingly difficult. While Prince's plan may seem idealistic, some believe that the overheated AI bubble could eventually burst, resetting the narrative and reining in the current wave of AI-fueled exploitation.