Growth Intelligence Brief #4

Cloudflare's crusade, Publishers fight back against AI Overviews, Substack + Yelp + Indeed in the SEO landscape check, How to project traffic and revenue for an e-commerce business and Content Snacks

Jul 10, 2025

∙ Paid

Welcome to another Growth Intelligence Brief, where organic growth leaders discover what matters - getting insights into the bigger picture and guidance on how to stay ahead of the competition.

As a free subscriber, you’re getting the first big story. Premium subscribers get the whole brief.

Today’s Growth Intelligence Brief went out to 398 marketing leaders (+22 since the last issue).

Here’s what’s covered this week:

Cloudflare’s crusade against rampage AI Crawlers
Publishers fight for survival with lawsuits
SEO landscape check (in focus: Substack, Yelp, Indeed)
Reader mailbag: how to project traffic and revenue for an e-commerce business
Content snacks

Cloudflare aims to create the first open-web content marketplace

The big shockwave over the last 2 weeks came from Cloudflare, which introduced Pay Per Crawl and announced the ability block AI Crawlers for new sites by default.

Here’s what happened:

Unless webmasters opt out of the new default block, AI Crawlers need to either pay the site owner or have permission to crawl the site.

AI crawlers will also need to state their purpose: training, search indexing, or reasoning.

So far, AI Crawlers went rampage and crawled as much as they could.

Most of them follow robots.txt rules (llms.txt… not so much) but Cloudflare’s new default might put an end to that and create a marketplace where creators get monetary reward for their content.

Why this news matters:

Cloudflare is HUGE: They handle ~80% of CDN traffic and manage ~20% of all web content based on their own statement. When such a giant changes the basic rules, it automatically has an impact.

This isn't just a minor change; it's the largest push yet to rebalance the balance sheet of the web.

Especially as especially as publishers are experiencing huge traffic losses under the new dynamics of searchers getting direct answers from AI (while rarely clicking through to sources).

AI crawlers often send a tiny fraction of traffic in relation to their crawls (see screenshot above). For every visit, OpenAI crawled a site on average 1,500 times - Perplexity 208 times and Google 9.2 times - over the last 12 months.

My take on this:

Google’s choice to display AI Overviews (and roll out AI Mode) broke the "generational contract" between search engines and content creators. While LLMs - like ChatGPT, Perplexity or Gemini - bypassed it right from the start.

Those who contribute the most high-quality content to the training data and search capabilities of LLMs get the least reward.

The content marketplace might be the most unbalanced, unfair, erratic, and unjust marketplace online today.

Central to the conversation is this question: What is a mention in an LLM answer worth?

Right now, I think a mention is worth much more than a click… but with two caveats:

The volume of mentions is much lower compared to clicks from search engines in the past
For publishers, the value of a mention is de facto 0

Revenue sharing for content has worked for Youtube but failed for Snapchat and TikTok.

Why?

YouTube’s ads are much easier to attribute to a creator’s video, ads integrate much better with long-form than short-form video, and Youtube built trust with advertisers and creators over decades.

Could this work on Cloudflare as well? It could, but only time will tell if LLMs are willing to shore up or not.

Here’s what to do:

Use Cloudflare to monitor AI crawls (Cloudflare dashboard -> AI Audit)
Scan the requests by Operator (AI Crawler) and decide for yourself which ones provide value to you vs no value at all
Block the AI Crawlers that don’t give you value (as a publisher, probably all of them)