BotNet
How the rapid growth of bots could change the open web
A warm welcome to 100 new Growth Memo readers who joined us last week! If you haven’t subscribed yet, join the ranks of Amazon, Microsoft, Google and 15,000 other Growth Memo readers.
Turn your blog into your biggest sales engine.
How? Focus on high-impact SEO strategies that target the most profitable opportunities.
North Star Inbound has a proven track record of creating top-converting blog pages that deliver
thousands of leads,
hundreds of thousands in directly-attributed revenue,
and rapid payback periods on SEO spend.
They focus on the 10% of pages with the highest ROI for your brand. Basically, your content becomes your most effective sales tool.
There’s no better way to loosen tight budgets than to grow revenue quickly.
Intrigued? Learn more here
How does the rapid growth of bots change the open web?
More than half of the web’s traffic comes from bots:
Imperva’s Bad Bot Report 2024 reveals that almost 50% of total internet traffic wasn’t human in 2024. Trend: declining.1
Cloudflare’s Radar shows ~70% human traffic and 30% from bots.2
Akamai reports that 42% of traffic on the web comes from bots.3
So far, bots have collected information that makes apps better for humans. But a new species grows in population: agentic bots. For two decades, we optimized sites for GoogleBot. Soon, we could focus on AI helpers that act as intermediaries between humans and the open web.
We already optimize for the BotNet with Schema and product feeds in Google’s Merchant Center. XML sitemaps have become table stakes decades ago. At the next level, we could have separate websites or APIs for agentic bots with a whole new marketing playing field.
Agentic web
As we’re getting close to getting more bot than human traffic on the open web, it’s essential to keep in mind that about 65% of bot traffic is estimated to be malicious. Good bots include scrapers from search engines, SEO tools, security defense and, of course, AI crawlers. Both types of bots are growing in population, but only the good ones are really useful.
Comparing the last with the previous 6 months, AI crawlers grew by 18% while search engines slowed down by 10%. GoogleBot specifically reduced activity by -1.6%. However, Google’s AI crawler made up for it with an increase of 1.4%. GPT Bot was the most active AI crawler, with 3.8% of all requests and grew 12%. Google’s AI crawler grew by 62% and was responsible for 3.7% of all bot requests. Given its current growth rate, Google’s AI crawler should soon be the most active on the web.4
Today, AI bots have three goals:
Collect training data
Build a search index to ground LLM answers (RAG)
Collect realtime data for prompts that demand freshness
But currently, all big AI developers work on agents that browse the web and take action for users:
Claude was first with its “Computer Use” product feature: “Developers can direct Claude to use computers the way people do—by looking at a screen, moving a cursor, clicking buttons, and typing text.”5
Google’s Jarvis, “a helpful companion that surfs the web for you”, accidentally launched on the Chrome store momentarily.6
OpenAI works on “Operator”, also an agent that takes action for you.7
I see three possible outcomes:
Agents either grow bot traffic on the open web significantly as they crawl and visit websites
Agents use APIs to get information
Agents operate on their platform, meaning Operator uses only data from Chat GPT instead of collecting its own
I think a mix of all three is most likely, with the result that bot traffic would grow significantly. If that turns out to be true, companies will increasingly build separate versions of their site for bots geared towards speed and structured data. “Just focus on the user” becomes “just focus on the agents”.
In a way, Chat GPT Search is already an agent that browses and curates the web for humans. But if agents keep their promise, you can expect a lot more. OpenAI CEO Sam Altman in his recent Reddit AMA: "I think the thing that will feel like the next giant breakthrough will be agents.”
As bots conquer the web, what are humans doing?
Internet adoption
It’s unlikely that humans stop browsing the web completely. Even with AI answers in Search and Search features in AI Chatbots, humans still want to verify AI statements, get randomly inspired (serendipity) or seek answers from other humans (Reddit). But the act of browsing to search will likely dissolve into prompts if bots get good enough.
There is room for growth in human traffic: 70% of the world’s population had internet access in 2023. At the current pace (~7% Y/Y), the whole world would have internet access by 2030. However, despite growing internet adoption, human traffic has been flat across the last 3 years (see Cloudflare stats). AI crawlers grow much faster (18%), and agents might speed it up even more.
Human requests to Wikipedia, the web’s largest site, have stagnated since 2019.8
The reason is that human attention shifts to social platforms, especially for younger generations. As the bots take the open web, humans flee to engagement retreats.
Marketing
In the distant future, good bots could become equal rights citizens of the open web nation as more humans spend time on closed social platforms and agentic LLMs increase the already fast growth of bot traffic. If this theory plays out, what does this mean for Marketing and specifically SEO?
Imagine booking a trip. Instead of browsing Google or Booking, you tell your agent where you think about going and when. Based on what it knows about your preferences, your agent curates 3 options of flights and hotels from your platform of choice. You pick a flight, it’s added to your calendar and the tickets are in your inbox. You don’t need to checkout. The agent does everything for you. You could apply the same scenario to e-commerce or software. Given that Companies like Google already have the capabilities to build this today, it’s worth thinking about what would remain constant, what would change and what would become more and less important in this vision.
Constant
Bots don’t need CSS or hero images. There is no downside to cloaking your site for LLM crawlers, so there is a chance that websites show bots a barebones version. A skillset that remains constant in this future is Technical SEO: crawlability, (server) speed, internal linking and structured data.
Change
An agentic open web offers much better ad targeting capabilities since bots know their owners inside out. Humans will make purchase decisions a lot faster since their agents give them all the information they need and know their preferences. Advertising costs drop significantly and offer even higher returns than already today.
Since bots can translate anything in seconds, localization and international sales are no longer a problem. Humans can buy from anyone, anywhere, only constraints by shipping and inventory. The global economy opens up even further.
If we play it right, agents could be the ultimate safe keepers of privacy: no one has as much data about you as they do, but you can control how much they share. Agents know you but don’t need to share that information with others. We might share even more data with them, spinning the value flywheel of data → understanding → results → data → value → results → etc.
On the downside, we need to build defenses against rogue bots by redefining what bots are allowed to do in a robots.txt 2.0-like format. Cybersecurity becomes even more important but also more complex since bad bots can mimic good ones so much better.
We’ll need to figure out the environmental impact of more energy consumption by higher bot traffic. Hopefully, bots are more efficient and, therefore, cause less total web traffic than humans. This would at least somewhat offset the energy gorge LLMs are already causing.
Important
The most bot-friendly format of information is raw and structured: XML, RSS and API feeds. We have already sent product feeds and XML sitemaps to Google, but agents will want more. Web design will be less important in the future, and maybe that’s okay since most sites look very similar anyway. Feed design becomes more important: what information to include in feeds, how much, how often to update it and what requests to send back to bots.
Marketers will spend a lot more time reverse-engineering what conversations with chatbots look like. Bots will likely be a black box like Google’s algorithm, but advertising could bring light into what people ask the most.
Relationships
In the agentic future, it’s hard to get customers to switch once customers have settled on a brand they like until they have a bad experience. As a result, an important marketing lever will be getting customers to try your brand with campaigns like discounts and exclusive offers. Once you have conviction and signal that your product is better, the quest becomes persuading users to try it out. Obviously, that already works today. With so many opportunities to advertise and sway users towards a specific product organically before they buy, we have a lot more influence on the purchase. But, in the future, agents could make those choices for users. Marketers will spend more time on relationship building, building brand awareness and influencing oldschool marketing factors like pricing, distribution (shipping) and differentiation.
Hey, what role do you see online paid advertising (paid search and others except for maybe social) having in the context of agentic future?
I think the key factor in this scenario will be trust and thats where I see paid advertising at a clear disadvantage. Paid advertising intrinsically makes up the lack of "trust"/"curation" by having the most attention/best search space etc.
If a human fully trusts that the purchase recommendations of the agent is the one that maximizes its value, I dont see how paying for a premium spot will compensate.
Another thing, is that it seems like most agents might up monetizing via a subscription.
Very good insight 😌. Can I translate part of this article into Spanish with links to you and a descripción of your newsletter?