I wonder if there is a counter-analysis that can be made with the same data though.
Your data finds that, compared to AI chatbots, Google search traffic A) has a shorter session duration and B) hits fewer pages.
This post takes that to indicate less engagement and therefore less conversion. But what if those data points actually indicate *higher user satisfaction* and therefore higher conversion?
Users want to find what they want *fast*, and happy users are more likely to convert. What if more time on site is actually bad for users--and conversion?
Perhaps I missed it, but I didn't see answers to my questions in the data. If I did miss it, please let me know!
Great point. I had this in my draft but then took it out. Long story short, the fact that AI Chatbots get more non-bounced pageviews in combination with higher session duration tells me that they take more action on sites compared to when they come from Google.
Working with websites in the software development services niche, I can confirm that traffic from AI platforms tends to have a higher conversion rate - perhaps because users trust AI as an advisor.
I’ve seen a similar effect with independent listings that feature companies, such as “top software development companies in the USA”.
It looks like the data suggests that AI chatbot traffic is more engaged and potentially more valuable than traditional search traffic for certain types of websites, especially as users engage in more conversational queries.
Do you think this shift towards conversational AI will continue to reshape SEO, or its too early to tell?
Unless I am missing something, this analysis raises significant concerns about data reliability and methodology. Addressing just a few:
Firstly, the 7M session dataset from Similarweb lacks transparency: there's no breakdown of how those sessions are distributed across the different AI chatbots (ChatGPT, Copilot, Gemini, Perplexity) and Google Search. This makes it impossible to assess the data's representativeness and potential biases. By your own admission AI chatbot traffic is 'tiny' (<1%) compared to Google and should immediately cast doubt on the validity of drawing broad conclusions. This vast difference in scale makes any comparison between AI chatbot user behavior and search engines inherently unreliable and misleading. Lastly, this is intended to focus “on transactional referral traffic”. How? Is it decided based on the page they landed on, based on a search query?
Chart 1 (AI Chatbot Forecast) is problematic. Using raw traffic numbers instead of percentages is highly unusual for this type of forecast, which typically expresses projected traffic share (e.g., "X% of traffic"). The data source, "six B2B companies," is vague. Where did this data originate? Is it aggregated, averaged, and does it represent monthly or annual figures? The "Now" data point lacks clarity. Finally, the chart's depicted dips and sudden jump are illogical. AI traffic share is expected to grow gradually, not spike erratically on top of existing search volume.
Additionally, the presence of duplicate entries (e.g., parent.netnany.com under Copilot, amazon.com under ChatGPT, Gemini, and Perplexity) indicates a lack of proper data cleaning, a vital step to any data analysis.
1/ Since everything is a percentage share or average within the referral source, nothing should be skewed by the relative share of data by platform. Also, Similarweb asked me not to share the breakdown so as to not give away too much to competitors.
Hi Kevin. Thanks for the quick reply, my follow-ups
1/ Since everything is a percentage share or average within the referral source, nothing should be skewed by the relative share of data by platform.
If your dataset comprises less ~1% of AI chatbot's traffic and ~99% of Google Search traffic, then the analysis is inherently skewed. You're essentially comparing a broad average from Google with niche segments from the AI chatbots. This means Google's data represents general user behavior across a massive population, while the AI data reflects the behavior of small, potentially unrepresentative groups. This disproportionate representation undermines any comparison between AI chatbot traffic and Google Search traffic. In other words, it's likely that similar niche segments exist within Google's 99% of the traffic, but they are masked by the vastness and diversity of the overall Google user base.
2/ You can find answers to your questions in the article I linked above the chart:
This information clarifies some aspects but raises further concerns. While expressing ratios in percentages in your text is sensible, the chart still uses numbers, making interpretation difficult. The inconsistency in timeframes is also problematic: your text refers to 'Today' as January to November, while the chart specifies June to November 2024 as the source.
Most importantly, the chart's source is stated as “6 B2B companies,” yet you admit to extrapolating data for Year 3 based on personal projections. This is a direct contradiction. Presenting speculative projections alongside actual data from a small sample size is misleading.. It creates the impression of a data-driven projection when it's heavily influenced by personal assumptions.
3/ Regarding Table 1, it seems the data doesn't agree with your expectations. That doesn't make the study unreliable.
Dismissing my critique as merely not meeting expectations overlooks the objective issues I've identified within the data itself. While my interpretation of these issues might involve some subjective judgment, the inconsistencies and contradictions in the data are factual.
A 7 million session dataset featuring a deep link to the Microsoft Outlook/Hotmail page at #4 behind Amazon, Ebay, and Home Depot raises serious questions about the data's representativeness. (e.g., searches for "hotmail" are overrepresented).
It's highly unusual for such niche pages to appear alongside major retailers in a large dataset. This outcome defies common sense and suggests potential flaws in data collection or sampling. It's like claiming that a survey of favorite foods among a large population resulted in kale chips topping the list alongside pizza and burgers. Such a result would immediately raise red flags.
Additionally, the presence of duplicate Landing pages for the same source underscores a lack of proper data cleaning, a fundamental step in any credible analysis. (e.g., how is amazon.com #1 and #4 on ChatGPT).
1/ Understood, but the differences are not extreme.
2/ I've been very clear about where the data comes from and how to interpret it. I haven't claimed that it's universally representative, but I shared it to make the point that AI Chatbot referrals are still small compared to organic traffic. Where is this contradicting? BTW, the extrapolations weren't "personal" but based on the current trend.
3/ Sure, it may be unusual, but the data is the data. Filtering data out due to personal preferences or expectations seems to be the opposite of what you're advocating for here. Regarding the duplicate landing pages, they had different UTM parameters that I took out to make the chart more readable. Doesn't change the outcome, though.
I'm still a bit confused about the Year 3 projection, though. You mentioned it's based on a "hunch," which seems like a personal assumption rather than a trend derived from the 6 B2B companies.
Since the chart clearly attributes the data to those companies, including a projection not directly derived from their data is the contradiction (source data is inconsistent). Perhaps a clearer source distinction in the chart would enhance transparency and ensure accurate interpretation.
3/
The premise of your article is AI converts better, and that the "data focuses on transactional referral traffic." Yet, to me, your data is conveying a fairly large chunk of non-transactional data -- I'm not sure there is another explanation for the MS Outlook/Hotmail, Amazon Prime, Amazon Videos positioning here for Google Search, outside of branded searches for "hotmail", "amazon prime" or "amazon videos", which I would argue are more navigational branded searches than having transactional intent.
To accurately test your hypothesis that AI converts better, the dataset should be strictly transactional. This isn't about filtering data based on personal preferences, but about ensuring the data is relevant to the research question. A flawed dataset will lead to flawed conclusions.
If you're studying if basketball players are taller than the general population, you wouldn't leave jockeys in your dataset. It's not about excluding jockeys because of "personal preferences or expectations ", but because they aren't relevant and would skew the data.
Probably not, because LLMs cover so much informational search. I'd rather look at brand mentions for informational search. Trying to find some data :).
Very rich breakdown here!
I wonder if there is a counter-analysis that can be made with the same data though.
Your data finds that, compared to AI chatbots, Google search traffic A) has a shorter session duration and B) hits fewer pages.
This post takes that to indicate less engagement and therefore less conversion. But what if those data points actually indicate *higher user satisfaction* and therefore higher conversion?
Users want to find what they want *fast*, and happy users are more likely to convert. What if more time on site is actually bad for users--and conversion?
Perhaps I missed it, but I didn't see answers to my questions in the data. If I did miss it, please let me know!
I was thinking the same thing. Great point. As always, more data is needed.
Thank you, Nate!
Great point. I had this in my draft but then took it out. Long story short, the fact that AI Chatbots get more non-bounced pageviews in combination with higher session duration tells me that they take more action on sites compared to when they come from Google.
Sure I see that thought process for sure...
But couldn't it also hint that users are less satisfied, have more difficulty finding what they need or want?
It just seems inconclusive to me......
Yes, we absolutely need more data to stress test this observation and make the argument more robust.
Working with websites in the software development services niche, I can confirm that traffic from AI platforms tends to have a higher conversion rate - perhaps because users trust AI as an advisor.
I’ve seen a similar effect with independent listings that feature companies, such as “top software development companies in the USA”.
Epic analysis Kevin!
🙇♂️thank you!
Solid article Kevin.
It looks like the data suggests that AI chatbot traffic is more engaged and potentially more valuable than traditional search traffic for certain types of websites, especially as users engage in more conversational queries.
Do you think this shift towards conversational AI will continue to reshape SEO, or its too early to tell?
Thank you! I think it's already reshaping SEO in many ways :). What do you think?
Really good stuff Kevin!
Thank you, Harry!
Hi:
Unless I am missing something, this analysis raises significant concerns about data reliability and methodology. Addressing just a few:
Firstly, the 7M session dataset from Similarweb lacks transparency: there's no breakdown of how those sessions are distributed across the different AI chatbots (ChatGPT, Copilot, Gemini, Perplexity) and Google Search. This makes it impossible to assess the data's representativeness and potential biases. By your own admission AI chatbot traffic is 'tiny' (<1%) compared to Google and should immediately cast doubt on the validity of drawing broad conclusions. This vast difference in scale makes any comparison between AI chatbot user behavior and search engines inherently unreliable and misleading. Lastly, this is intended to focus “on transactional referral traffic”. How? Is it decided based on the page they landed on, based on a search query?
Chart 1 (AI Chatbot Forecast) is problematic. Using raw traffic numbers instead of percentages is highly unusual for this type of forecast, which typically expresses projected traffic share (e.g., "X% of traffic"). The data source, "six B2B companies," is vague. Where did this data originate? Is it aggregated, averaged, and does it represent monthly or annual figures? The "Now" data point lacks clarity. Finally, the chart's depicted dips and sudden jump are illogical. AI traffic share is expected to grow gradually, not spike erratically on top of existing search volume.
Table 1 (Top 10 LPs): This table suggests either a skewed sample, or a flaw in the data collection or analysis methodology. One example: I would not expect a truly representative dataset of 7M sessions would feature a deep link to: https://www.microsoft.com/en-us/microsoft-365/outlook/email-and-calendar-software-microsoft-outlook (unless, 99% of the dataset is on email and calendar software).
Additionally, the presence of duplicate entries (e.g., parent.netnany.com under Copilot, amazon.com under ChatGPT, Gemini, and Perplexity) indicates a lack of proper data cleaning, a vital step to any data analysis.
Hi! Thanks for sharing your concerns.
1/ Since everything is a percentage share or average within the referral source, nothing should be skewed by the relative share of data by platform. Also, Similarweb asked me not to share the breakdown so as to not give away too much to competitors.
2/ You can find answers to your questions in the article I linked above the chart: https://www.growth-memo.com/p/how-significant-is-ai-chatbot-traffic
3/ Regarding Table 1, it seems the data doesn't agree with your expectations. That doesn't make the study unreliable.
Hi Kevin. Thanks for the quick reply, my follow-ups
1/ Since everything is a percentage share or average within the referral source, nothing should be skewed by the relative share of data by platform.
If your dataset comprises less ~1% of AI chatbot's traffic and ~99% of Google Search traffic, then the analysis is inherently skewed. You're essentially comparing a broad average from Google with niche segments from the AI chatbots. This means Google's data represents general user behavior across a massive population, while the AI data reflects the behavior of small, potentially unrepresentative groups. This disproportionate representation undermines any comparison between AI chatbot traffic and Google Search traffic. In other words, it's likely that similar niche segments exist within Google's 99% of the traffic, but they are masked by the vastness and diversity of the overall Google user base.
2/ You can find answers to your questions in the article I linked above the chart:
https://www.growth-memo.com/p/how-significant-is-ai-chatbot-traffic
This information clarifies some aspects but raises further concerns. While expressing ratios in percentages in your text is sensible, the chart still uses numbers, making interpretation difficult. The inconsistency in timeframes is also problematic: your text refers to 'Today' as January to November, while the chart specifies June to November 2024 as the source.
Most importantly, the chart's source is stated as “6 B2B companies,” yet you admit to extrapolating data for Year 3 based on personal projections. This is a direct contradiction. Presenting speculative projections alongside actual data from a small sample size is misleading.. It creates the impression of a data-driven projection when it's heavily influenced by personal assumptions.
3/ Regarding Table 1, it seems the data doesn't agree with your expectations. That doesn't make the study unreliable.
Dismissing my critique as merely not meeting expectations overlooks the objective issues I've identified within the data itself. While my interpretation of these issues might involve some subjective judgment, the inconsistencies and contradictions in the data are factual.
A 7 million session dataset featuring a deep link to the Microsoft Outlook/Hotmail page at #4 behind Amazon, Ebay, and Home Depot raises serious questions about the data's representativeness. (e.g., searches for "hotmail" are overrepresented).
It's highly unusual for such niche pages to appear alongside major retailers in a large dataset. This outcome defies common sense and suggests potential flaws in data collection or sampling. It's like claiming that a survey of favorite foods among a large population resulted in kale chips topping the list alongside pizza and burgers. Such a result would immediately raise red flags.
Additionally, the presence of duplicate Landing pages for the same source underscores a lack of proper data cleaning, a fundamental step in any credible analysis. (e.g., how is amazon.com #1 and #4 on ChatGPT).
1/ Understood, but the differences are not extreme.
2/ I've been very clear about where the data comes from and how to interpret it. I haven't claimed that it's universally representative, but I shared it to make the point that AI Chatbot referrals are still small compared to organic traffic. Where is this contradicting? BTW, the extrapolations weren't "personal" but based on the current trend.
3/ Sure, it may be unusual, but the data is the data. Filtering data out due to personal preferences or expectations seems to be the opposite of what you're advocating for here. Regarding the duplicate landing pages, they had different UTM parameters that I took out to make the chart more readable. Doesn't change the outcome, though.
2/
I'm still a bit confused about the Year 3 projection, though. You mentioned it's based on a "hunch," which seems like a personal assumption rather than a trend derived from the 6 B2B companies.
Since the chart clearly attributes the data to those companies, including a projection not directly derived from their data is the contradiction (source data is inconsistent). Perhaps a clearer source distinction in the chart would enhance transparency and ensure accurate interpretation.
3/
The premise of your article is AI converts better, and that the "data focuses on transactional referral traffic." Yet, to me, your data is conveying a fairly large chunk of non-transactional data -- I'm not sure there is another explanation for the MS Outlook/Hotmail, Amazon Prime, Amazon Videos positioning here for Google Search, outside of branded searches for "hotmail", "amazon prime" or "amazon videos", which I would argue are more navigational branded searches than having transactional intent.
To accurately test your hypothesis that AI converts better, the dataset should be strictly transactional. This isn't about filtering data based on personal preferences, but about ensuring the data is relevant to the research question. A flawed dataset will lead to flawed conclusions.
If you're studying if basketball players are taller than the general population, you wouldn't leave jockeys in your dataset. It's not about excluding jockeys because of "personal preferences or expectations ", but because they aren't relevant and would skew the data.
This is fascinating, thanks for the insight.
Are you planning on doing a similar analysis for informational search?
Probably not, because LLMs cover so much informational search. I'd rather look at brand mentions for informational search. Trying to find some data :).