Cloudflare CEO warns AI crawlers and summaries are eroding the internet's business model

[Daniel Sims]

In context: Cloudflare CEO Matthew Prince recently reiterated his warning that generative AI crawlers and summaries threaten the foundations of the internet's business model. To protect publishers from a flood of artificial AI traffic that offers virtually no authentic site visits in return, the company is devising methods to combat AI scrapers.

Speaking at an Axios event in Cannes last week, Prince explained that search engines and chatbots using generative AI to summarize web content have significantly reduced the number of human visitors to many websites. Even compared to six months ago, the problem has worsened considerably.

Traditionally, for every six times Google crawled a website, one person might visit and potentially view ads. In contrast, the rate was about 250 to 1 with OpenAI's crawlers and 6,000 to 1 with Anthropic. Today, Cloudflare's CEO estimates that Google's crawl-to-visitor rate has declined to 18 to 1, OpenAI's has worsened to 1,500 to 1, and Anthropic's is approximately 60,000 to 1.

This decline is likely because chatbots and search engines now save users the effort of visiting websites. Chatbots can retrieve and summarize information without the user needing to leave the chat interface, and AI overviews from major search engines now offer users answers before they click on search results.

When Google first unveiled AI overviews, it claimed that the technology would boost traffic to the original sources of the summarized content. Similarly, large language models such as ChatGPT have recently started citing sources in their responses to help direct traffic back to content creators.

Must read: The Zero Click Internet

However, Prince claims that users, by and large, aren't clicking on the footnotes. Instead, many many are accepting the AI's responses at face value, as trust in the technology has grown over the past six months. Aside from starving websites of traffic and revenue, the trend is potentially dangerous due to AI's known tendency to generate inaccurate or misleading information.

In response, Cloudflare, which offers cybersecurity solutions for websites, has launched a new tool called the AI Labyrinth. This tool is designed to use generative AI against the crawlers themselves.

Although websites can include instructions to block AI crawlers, many bots either bypass or ignore these directives. When the AI Labyrinth detects such behavior, it leads the bot through a maze of AI-generated links that no human would reasonably follow, causing the bot to waste time and computing resources.

Despite how daunting it might seem to oppose AI giants like Google, Microsoft, and OpenAI, Prince emphasizes that Cloudflare has a strong track record of successfully defending its clients, even against attacks from powerful national governments.

Permalink to story:

Cloudflare CEO warns AI crawlers and summaries are eroding the internet's business model

 

[NumberNine]

The rich people are crying again.

 

[trparky]

The rich people are crying again.

Let them… I don’t care.

 

[Squid Surprise]

The internet's business model was fundamentally broken to begin with. Finally, something good from AI

 

[brucek]

Ad blocking worked before, and still works now. No change.

Quick queries seeking simple facts ("What temperature does water boil") may get an immediate AI answer now. They previously already received a non-AI google answer. Before that they may have linked to a 3rd party site but either way there was not going to be any engagement from me. I'd see the answer and close the page.

Yet most of my browsing remains outside of Google & AI on sites I have engagement with, such as TechSpot. I view these sites by direct bookmark or muscle memory, not by being linked. That seems unlikely to change.

 

[mbk34]

I guess the choice for users is to either disbelieve the AI summary or to click through pages of cookie options and then disbelieve the long version with yet more adds. This was all forecast to happen when these AI summaries started appearing, I think there was an excellent article on here about a year ago. Maybe these CEO's just don't keep up?

 

[TooQik]

The internet as it exists itself does not have a business model, it exists purely as a means of inter-connectivity. It's individuals and companies that chose to utilise the internet infrastructure to conduct business, that will implement business models as they see fit.

If AI changes the way search engines present website content, then those that rely on advertising revenue to fund their online presence will need to change their business model to accommodate. I expect that this will have three flow on effects. First, some sites will chose to fund the activities in other ways eg. put them behind pay-walls. Next, some sites will simply shut down. Last, advertising placement will move from individual websites to search engine front pages, which I'm sure is exactly what companies like Google are pushing for.

 

[wiyosaya]

Surprise, surprise! Dumb piece of software marketed as Intelligent degrades the internet amidst growing number of companies eschewing its use because they figured out they have been duped by marketing.

Seems more and more people are figuring out that AI is a worthless POS in most cases. In cases where it actually gives useful information, it is highly monitored and tailored by humans. Think Medical.

 

[BuckarooBonzaii]

Are there no more Honor Among Thieves? As always money claims the victor.