Google vs ChatGPT

Which one do people prefer for search?

Research by digital marketing consultancy ClickPop explores people’s preferences for website recommendations provided by Google vs ChatGPT. 

Spoiler alert – most people preferred ChatGPT.

Quick links

• Results
• The differences between the websites returned by Google and ChatGPT
• Possible implications for the future of search
• About the study

Introduction

To test whether people prefer website recommendations from Google or ChatGPT, ClickPop created a tool where users could simultaneously search Google and ChatGPT, and then display the two sets of website recommendations in separate, randomised sets; A or B. The user could then vote for which set of websites was a better match for their search. Think of it like the Pepsi Challenge but between Google and ChatGPT.

The test ran from 29 May 2023 to 19 July 2023. In total, 1,674 searches were carried out with 726 votes being placed.

Interestingly, the preference for ChatGPT reduced only slightly when the user made a search that specified a local area, for example, “campsites in North Cornwall”, or “Hotels in Deventer”. For these searches, 61.1% of people preferred the websites recommended by ChatGPT, however, this is based on a small number of queries (18).   

To create a fair test, whether Google or ChatGPT’s results were displayed as Set A or Set B was randomised and presented to the user anonymously. Google’s results were shown in Set A 369 times, and ChatGPT’s were shown in Set A 357 times. Whether Google’s or ChatGPT’s results were shown in Set A or Set B made little difference to the votes, see table.

Table 1 – Votes for Google or ChatGPT by set

Similarly, the device people used to conduct the search made no significant difference to their preference, table 2 below.

Table 2 – Votes for Google or ChatGPT by device

What are the differences between the results?

Google provides slightly more variety

Of the websites provided by Google and ChatGPT, Google provided slightly more variety in its results giving 2,597 different websites across all searches, compared to 2,457 from ChatGPT.

But Google gave fewer results from retailers 

Of the top 20 most frequently provided websites by Google, 11 of the 20 were retailers, brands or websites that facilitate a purchase. 9 of the 20 most frequently recommended websites from Google were publishers, directories, social networks or product comparison websites. 

Of the 20 most frequently recommended websites provided by ChatGPT, 19 were retailers, brands or websites that facilitate a purchase, and only 1 was a website where you couldn’t purchase directly from them (e.g. publishers, directories, social networks or product comparison websites). 

​​Table 3 – 20 most frequently provided websites from Google

Table 4 – 20 most frequently provided websites from ChatGPT

Differences in Domain Authority

Analysing the top 200 most frequently mentioned websites for both Google and ChatGPT shows some interesting differences in the authoritativeness of the websites. Using SEMRush’s Domain Authority for the top 200 most frequently mentioned websites we plotted the results on a scatter graph, see below

The above chart shows that there is a trend for both Google and ChatGPT to return websites with the highest domain authority more frequently than websites with lower domain authorities. However, Google returns websites with medium to low domain authorities more frequently than ChatGPT. This is emphasised in the below chart which groups the domain authorities into brackets.

What might this all mean?

While the limitations of the tool mean that we’re not able to compare the real-world experience of using Google or ChatGPT, the fact that most people preferred the websites recommended by ChatGPT in a randomised, blind test is interesting. 

When conducting their search, users were instructed to search for a product or service. With the tool amending the search query entered by the user into a query directing Google and ChatGPT to find “websites that provide [query] in [country]”. This amendment should have meant that both Google and ChatGPT were left in no doubt that this is a search conducted by someone wanting to make a purchase. However, Google frequently returned websites where it’s not possible to make a purchase and instead ranked websites with a high domain authority but are publishers.

Possible outcome #1: In the future Google will show fewer product comparison and publisher websites for commercially relevant search terms, regardless of their perceived authority.

 

 

Possible outcome #2:  Instead of making assumptions about the user’s search intent, Google may display more results but group results into categories where users can scroll through them depending on their intent. While this is counter to recent trends of Google reducing the number of organic results in favour of more real estate for paid results, it’s worth noting that Google was able to make those changes while it benefited from holding a near monopoly on search.

When looking at the top 200 most frequently returned websites, ChatGPT recommended more websites with a domain authority of between 51 and 80 than Google, and fewer websites with a domain authority of 50 or less.  This may be because, with more limited resources, ChatGPT has focused its data collection on capturing information about the most popular and prominent sites. This may change in future as ChatGPT’s resources increase. However, with so many people having a preference for the results provided by ChatGPT, it may be that users simply have a preference for recognisable brands that they know and trust.

Possible outcome #3: In the future Google will include fewer websites with medium to low domain authority in its first-page results.

 

About the study

The study was carried out between 29 May and 19 July 2023. In total, 1,674 searches were conducted, and 726 votes were made using this tool.

The tool we created lets people search for products or services and simultaneously fetches the top ten websites provided by Google and ChatGPT.

It then presents these results in two anonymised sets (A and B) to the searcher. To make it a fair test, the tool randomises whether Google’s or ChatGPT’s results appear in Set A or Set B.

Having been presented with the two sets of anonymous results, the searcher can then click on any of the websites presented, and then vote on which set of websites they feel are the better results for their search. The user is then told whether it was Google or ChatGPT that provided their results.

When a visitor searches using the Google vs ChatGPT tool, it simultaneously searches Google and ChatGPT (using GPT 3.5, Turbo) for the results. 

With Google, the tool identifies the searcher’s country based on their IP address and searches the appropriate country version of Google. For instance, if someone searches from the UK, the search will be carried out using Google UK. That way the results will be from websites that serve the UK and not always from companies that predominantly serve the US. The tool also uses the searcher’s country for the ChatGPT search but in a slightly different way.

The tool searches Google and ChatGPT in different ways. For Google, the tool amends the text entered into it to query Google in the following way: “websites that provide (WHATEVER THE QUERY IS) in (WHATEVER THE COUNTRY IS)”. The reason for this is that with most searches, Google will have to make assumptions about the intent of the search. For example, if someone searches “Air Fryers”, Google will not necessarily be sure whether the person wants to buy an air fryer, find out information about air fryers, see product reviews etc. By amending the search in the following way, the intention is that Google should be left in no doubt that the search is intended to make a purchase. In addition, this alteration makes for a more fair comparison to the search query and results provided by ChatGPT, see below. 

ChatGPT is an AI that uses natural language processing. This means it needs more context to the product or service being searched for than simply the product’s generic name. When the user searches using the Google vs ChatGPT tool, it prompts ChatGPT by adding some additional context to the search by dropping it into the following sentence: “Show me links to the top 10 websites that provide (WHATEVER THE QUERY IS)  in (WHATEVER THE COUNTRY IS)”. Using the earlier “Air Fryers” example from someone based in Australia, the tool will query ChatGPT with the following prompt: Show me links to the top 10 websites that provide Air Fryers in Australia.

There are, however, some significant limitations of this tool: 

• ChatGPT’s knowledge only goes up to September 2021 so the results it provides are not as up-to-date as Google’s.

• When recommending websites for a query, ChatGPT often doesn’t link to a specific page on the website, but will instead provide the home page. For this reason, all search results using this tool are curtailed to just show the home page whether the results are from Google or ChatGPT. This helps keep it a fair comparison, however, it isn’t representative of the real-world experience of using Google.

• The comparison tool also only shows the website address and none of the associated metadata that you would expect to see in a list of results provided if you were searching Google.

• Sometimes, ChatGPT will not be able to recommend any websites for the query. When this happens the user will just see a message saying their search was a bit obscure and to try something else. 

People from 84 different countries made searches using this tool, see below data.