11 realizations from testing Search Generative Experience
11 realizations from testing Search Generative Experience
I spent countless hours stress testing Google's new AI-powered search experience SGE (Search Generative Experience) since it opened access last week.
In the process, I had 11 realizations about how we can reverse-engineer AI results, optimize for SGE and estimate the impact.
While SGE is still in beta and has many flaws, many of the assumptions I wrote about in The future of SEO hold true: the search landscape changes significantly and pushes us to face new problems and learn new skills.
How Google might make SGE cost-efficient
The first realization is that Google doesn’t show AI results by default, at least in the SGE beta. Instead, users need to hit a “generate” most of the time (see screenshot below).
The generate button seems like a detail, but it makes a big difference.
By not showing AI results by default, Google can test engagement (do users really want AI results?) and save money.
While I am personally convinced that AI answers are inevitable in the medium term to long term, it’s not yet clear whether users prefer AI answers over classic web results in the short term. The “generate” button lets Google measure how interested users are in AI results and test engagement (I noticed clicking it will expand AI Snapshots by default in my next searches).
Generative AI results are much more expensive than classic web results. In a Reuters interview, Alphabet chairman John Hennessy projected the potential cost increase per query at 10x.
Google, for instance, could face a $6-billion hike in expenses by 2024 if ChatGPT-like AI were to handle half the queries it receives with 50-word answers, analysts projected. [link]
And yet, I don’t think the cost argument is as big as many think. That's my second realization.
First, Alphabet spent $208b and made $60 in profits in 2022. Another $6b would be a mere +2.8% increase in expenses - peanuts for a behemoth!
Second, not every query needs AI answers. Navigational queries to websites just need a link, even though SGE popped up for a few brand names. Simple searches that yield direct answers today don't need AI. As you'll notice, there are several cases in which the AI Snapshot offers way more than needed.
Third, Google might cache a significant amount of AI answers to save costs. An article by Arda Capital outlines how Google might be able to cache 30-60% of queries. Generative AI answers might be harder to cache than web results, but even a rate of 10-20% of lower cost significantly.
“Most analyses ignore the impact of caching. When someone searches “Youtube” in Google (one of the most popular queries), Google does not re-run that search query. It will show cached results, which helps reduce latency and compute costs. The historical cache hit rate for Google is between 30-60%. Google does not provide much public data on its search queries, but an analysis done on AOL’s search queries found that caching the top 1M queries captures ~60% of all queries.” Arda Capital [link]
Fourth, the cost of computing, renewable energy and AI models are all trending downward. In fact, a recently leaked document outlines how Open Source is a threat to Alphabet, Microsoft and OpenAI since models are cheap enough to evolve in a distributed way.
“While our models still hold a slight edge in terms of quality, the gap is closing astonishingly quickly. Open-source models are faster, more customizable, more private, and pound-for-pound more capable. They are doing things with $100 and 13B params that we struggle with at $10M and 540B. And they are doing so in weeks, not months.” Dylan Patel and Afzal Ahmad [link]
I assume “click to generate” is only temporary, and Google will ultimately display AI answers by default when cost and quality are good enough for queries where it makes sense.
How web results create generative AI answers in Search
My third realization is that we can reverse-engineer AI results.
Google displays a link carousel with websites next to AI answers (titled AI Snapshot), which likely gets the majority of remaining organic clicks moving forward. It's critical we understand how to rank in the carousel.
Google generates AI answers by grounding LLMs (large language models) in search results with a process called Retrieval augmented generation (RAG). Google’s corroboration feature lets us break AI answers down and see which web results contributed to what part. In plain terms, corroboration allows us to reverse engineer how Google generates AI answers.
When testing SGE, I learned that the sites shown in the AI Snapshot carousel are not the same links as shown in the classic search results. You would think Google forms AI answers out of the top Search results, but that's not the case. If AI Snapshot carousels get most organic clicks, but Google doesn't pick the best-ranking sites, we need to re-evaluate what signals Google uses to put the carousel together.
Here is my theory:
For the keyword “corporate credit card”, the first result in the AI Snapshot carousel ExpenseAnywhere, which ranks in position 26 of the classic search results. Again, ranking at the top of classic web Search doesn't mean you rank in the AI Snapshot carousel.
What seems to make ExpenseAnywhere the first carousel result is its unique angle: it’s the only result highlighting the risks of corporate credit cards.
SGE structures the AI answer around risks and benefits (or pros and cons) in his case. Corroboration shows that it pulls different information from several sites.
It’s unclear to me why SGE highlights benefits and risks instead of other aspects like important criteria when choosing a corporate credit card.
Google sends users to a specific passage when clicking one of the carousel links, using the same feature as for Featured Snippets. My fourth realization is that by explicitly writing about an angle AI Snapshots highlight, websites might increase the chance of ranking in the carousel.
Ranking in AI answers is a matter of a) understanding the different angles SGE covers and b) answering them explicitly in your content. We can look at the passage Google uses in Corroboration to get an understanding of what SGE is looking for and tune our content accordingly.
Ironically, Featured Snippets might not get traffic anymore when AI Snapshots are present.
I wouldn't be surprised if Google stopped showing Featured Snippets when AI Snapshots are present.
Ecommerce
My fifth realization is that AI answers in e-commerce are the most aggressive. For product queries (transactional intent) like “summer shirts for men”, Google shows two rows of 7 products on desktop and 2 products on mobile.
How Google decides which products to show isn’t clear to me at this point. They don’t overlap with products in the organic shopping carousel on the page.
For other queries, like “standing desk”, Google shows specific product recommendations based on attributes like “good for home office”, “good for tall people” or “good for gaming”.
Google seems to pull these product attributes from written reviews on websites, which is quite stunning because Google turns unstructured into structured data. I found “best for tall people” in reviews by BGR and Insider, which Google references in the “common uses” section in the right sidebar. My sixth realization is that product, local and brand reviews are gaining significant importance for AI results.
The right sidebar pops up when you click on a product in the AI Snapshot carousel. It features different retailers for the same product, which means a retailer offering a lower price can beat another retailer with better reviews or nicer snippets. Down the road, I expect Google to offer sponsored slots in this list and monetize product searches further.
Based on these observations, we can optimize the following signals for transactional intent queries:
3rd party reviews
Google reviews
Price
Thumbnail
Description
Attributes
Title
My seventh realization is that Google might offset higher costs of generative AI queries with more ads in the search results.
As always, organic results are massively influenced by ads. The product carousel in AI answers doesn’t show ads yet, presumably because SGE is still in beta. But Google has introduced what ads might look like in AI answers, and I expect them to share valuable slots with organic results [link]. Other than in the Google I/O demo, Shopping Ads appear below the AI Snapshot in the SGE beta, so we can already tell that Google is testing ad placement a lot.
Local Search
For local searches like “best sushi in chicago”, Google pulls the full local pack into AI results - and I'm not sure why. This is a case where I was surprised Google showed an AI Snapshot instead of classic search results.
Local results in the AI Snapshot have two differences from classic local packs. First, they try to give a much more tailored answer to the query. Businesses shown in the AI Snapshot are not the same as in classic local packs. Instead, SGE tries to customize the list based on reviews from sites like Yelp, Eater or Google itself (see screenshot below) based on what you search for.
SGE provides useful information like opening times (1), images from Google Maps (2), useful links (3) and reviews (4) for business names.
My eighth realization is sites that provide local reviews might actually get more traffic from Google based on the SGE beta, but there is a risk Google only links to its own reviews.
YMYL
Google doesn't show SGE for YMYL queries (Your Money or Your Life verticals like finance, health or legal) every time.
Queries like “should you get a personal loan to pay off credit card debt”, for example, don’t trigger an AI Snapshot.
However, “symptoms for diaper rash” triggers an AI Snapshot.
I assume Google stays away from giving specific advice for sensitive topics like loans or serious diseases, which are regulated in many countries. You can't simply give advice, and neither can Google.
I wasn’t able to figure out where the line for YMYL queries is. A query like “can essential oils cure cancer” returns a correct answer and even leverages research papers.
But keywords like “best credit card for poor credit score” or “best insurance for car” didn’t trigger SGE. I expect Google to be very careful with giving advice in regulated industries, which is good news for classic organic results.
My ninth realization is that AI answers for YMYL topics are very uncertain. I could see Google staying away from AI answers and leaning on its content partnerships - unless it's about treatments and medication, such queries aren't monetizable anyway.
Other design changes that fly under the radar
Apart from AI Snapshots, Google has launched a couple of design changes in the beta that might fly under the radar:
First, Titles are black instead of blue. I'm not sure it makes a big difference, but it stood out.
Second, query refinements and vertical search tabs seem to share the same space. The little bubbles under searches that push users into specific searches faster don't seem to appear in a specific order, except for the Converse tab always showing first.
I was very excited about the Add to spreadsheet feature, which allows users to add elements in Search to a spreadsheet. But to my disappointment, you can't add output from Bard to Sheets. Just websites. Why?
Verdict: certainly a change to search, but there might be a bigger opportunity for AI answers
SGE has some shortcomings. Not everything makes sense yet. Some features feel clunky. But it’s a beta, and unrefined features are a signal of shipping fast, which is critical for Google in my mind.
My tenth realization is that it's a lot harder for Google to fine-tune AI answers than classic search results. Before AI, Google was able to rearrange snippets and sometimes SERP features. Now, it needs to improve the angle, presentation, accuracy and diversity of answers, which is a lot more complicated.
I worry about more longtail searches. Since AI Snapshots are much better at answering longtail searches, users might ask Google a lot longer questions. At the same time, Google Search Console already hides a lot of data for long tail queries out of “privacy reasons''. If that dynamic doesn't change, we might see even less data from Google.
My eleventh realization is that the biggest opportunity for SGE is not in Google Search but in becoming an assistant for Google’s whole ecosystem.
This week, Microsoft introduced Windows Copilot, which combines the power of Chat GPT (+ plugins) with Bing Search, at its developer conference Build [link]. Users can execute actions, ask questions and perform searches right in the Operating System. If quality and speed are good enough, why should users visit another website to search?
Search would move to the OS. As a result, it wouldn’t only threaten Google Search but also make classic SEO data aggregation like SERP crawling impossible.
The bigger opportunity for Google is to break SGE out of google.com and integrate it with Chrome, Gmail, Youtube, Android, Pixel phones and all its other properties. Why should users still visit google.com if they could get Search in every app they use? Even better, SGE could be trained and improved based on the properties’ data, like emails in Gmail or specific parts of a video on Youtube.
Training generative AI models on user data would open the door for two trends that might have been ahead of their time: personalization and voice search.
Assistants like Siri, Alexa, Cortana or Google Assistant could become 10x, maybe 100x, better with generative AI. Voice search disappointed in the past but might become viable in the near future. Website and product personalization had a big hype about 10 years ago but never made it to a good enough level. Generative AI could gather enough data to understand the context necessary to truly personalize user experiences - especially with connected data lakes like Microsoft Fabric.
The challenge, as always for Google, is not killing its advertising marketplace. A personal assistant is much better suited for subscriptions than ads, which is why Microsoft is in a much better position to build an integrated SGE assistant. They already lost Search against Google, and I'm not confident AI will change that, but Google's weakness is its dependency on ad revenue. Microsoft can innovate much faster in this space because they don't need to make sure Search monetizes with ads. AI assistants are the actual frontier of the war.
But it will take time until this vision becomes reality. Until then, we'll have a lot more realizations about AI answers.