My hosting is blocking almost 90% of the LLM bot traffic. I wonder do you see the same in your server? And what’s your thought on this as LLM bot is known to be very aggressive. Do LLM bots need to be granted full access to our site for full visibility potential?
I'm not seeing the same with my clients, but 90% is severe. I wonder if a sensitive limiter is at play? If you want LLM traffic, LLM crawlers need full access to your site. The only other way is for LLMs to pick your content up through public indices like commoncrawl, but that's much harder to control and more likely to be incomplete and outdated.
Looks like it’s time to switch hosting then! I had spoken with their support and they are aware of that and they do not intend to lift that restrictions anytime soon.
Which tools would you recommend to help with tracking LLM crawlers and appearances in AI chatbots? At the minute is it just a case of checking GA for typical metrics or is there something more I could be doing?
I more or less suggest the same action items to the clients I talk to, so thanks for the sanity check.
Thanks for pointing out the law of compounding. It's so counterintuitive for humans and such small numbers are easy to dismiss. And like you write: Projections are often wrong but can show how the future COULD look like.
Then again: It's about way more than just referral traffic. If my target audience uses these tools a lot I would want to improve my brand's performance. YouTube, Reddit and other social channels might not send as much traffic, but there is no doubt about that they shape user perception.
My hosting is blocking almost 90% of the LLM bot traffic. I wonder do you see the same in your server? And what’s your thought on this as LLM bot is known to be very aggressive. Do LLM bots need to be granted full access to our site for full visibility potential?
I'm not seeing the same with my clients, but 90% is severe. I wonder if a sensitive limiter is at play? If you want LLM traffic, LLM crawlers need full access to your site. The only other way is for LLMs to pick your content up through public indices like commoncrawl, but that's much harder to control and more likely to be incomplete and outdated.
Looks like it’s time to switch hosting then! I had spoken with their support and they are aware of that and they do not intend to lift that restrictions anytime soon.
Specifically, my servers returned 520 to those AI bots.
woof, that sounds irresponsible to me...
Which tools would you recommend to help with tracking LLM crawlers and appearances in AI chatbots? At the minute is it just a case of checking GA for typical metrics or is there something more I could be doing?
For LLM crawls, you want to look at server log files and filter for LLM user agents.
For LLM visibility, you can lean on tools like Profoud or Scrunch AI.
I more or less suggest the same action items to the clients I talk to, so thanks for the sanity check.
Thanks for pointing out the law of compounding. It's so counterintuitive for humans and such small numbers are easy to dismiss. And like you write: Projections are often wrong but can show how the future COULD look like.
Then again: It's about way more than just referral traffic. If my target audience uses these tools a lot I would want to improve my brand's performance. YouTube, Reddit and other social channels might not send as much traffic, but there is no doubt about that they shape user perception.
Exactly! It's a mindset shift from referral traffic to engagement. Tough to make after decades of performance marketing, but necessary.