Google’s Changes do not Comply with Digital Markets Act

February29,2024

Google’sChangestoSearchResultsPagesdonot

ComplywithDigitalMarketsActsProhibitionon Self-Preferencing Introduction

As Europe’s Digital Markets Act (DMA), which aims to “make the markets in the digital sector fairer and more contestable” comes into effect on March 6. In anticipation of this deadline a number of designated “gatekeepers” have begun to signal their putative compliance plans in various forms, including public statements, workshops, and even pushing changes live on the internet.

Alphabet, the parent company of Google, is required to comply with rules to prevent it from abusing its market power which includes a prohibition on Google’s self preferencing its own services and products, under the DMA’s Article 6(5) Google now appears to be serving European users Search Engine Results Pages (SERPs) that are in line with a version of mock-ups it circulated to various parties, including Yelp, in December.

Yelp analyzed these mock-ups and determined that not only do they not meet Google’s obligations to cease self-preferencing they actually increase the rate at which users will remain within Google’s walled garden.

On its face, the SERPs Google has implemented continue to advantage Google’s inferior services and products above competitors’, in violation of Article 6(5). For example, the mapping features include links to Google services while excluding rivals, and Google services always appear in a variety of prominent modules even as any given rival appears at most some of the time. Moreover, Yelp’s analysis found that Google’s new European SERPs actually appear to exacerbate Google’s self-preferencing, leading it to retain even more traffic within its own properties than had its pre-DMA SERPs We found that under its pre-DMA SERPs Google retained approximately 55% of traffic. Under the new SERPs, Google retains approximately 73% of traffic.

To become compliant with the DMA, Google should be required to operate within a different paradigm: One in which it links to the best and most relevant content for users’ search queries — whether provided by Google or by a third-party service.

A Visual and Technical Analysis of Google’s (Noncompliant) DMA Plans

Google’s DMA plans appear to entail adding three new features to the SERPs and one new feature to the Entity Results Page. We will refer to the new SERP modules as the Vertical Search Service (VSS) results, the SERP “Chip” Filter Toggles, and the Comparison Sites module. The Entity Results Page module is a unit for a blend of both Google’s house product and third-party reviews with links to both.

Additionally, Google has removed the so-called “Commitments 3” unit that it had been serving to users in Europe since 2015

A Catalog of Google’s proposed SERP changes

NEW VSS RESULTS

This module replaces an organic link with a visual carousel with a horizontal scroll option. (This unit does not have the fixed position above the local OneBox that was implied in mock-ups Google circulated in December. In fact, it seems as though it appears in that position only rarely.)

NEW SERP FILTER TOGGLES

This unit allows a user to completely disable organic search and show only Places results (if “Places” is clicked) or disable the local OneBox (if “Comparison sites” is clicked). With the exception of “Comparison sites,” every toggle is an enhancement of Google’s “house” products (“Open now,” “Top-rated,” etc )

NEW COMPARISON SITES MODULE

This module appears mid-SERP and garners a low percentage of clicks due to its positioning on the page and less-direct relevance to a given query (i.e. serves up listicles in a horizontal scroll in response to queries where a user is searching for a specific type of business and is otherwise scrolling vertically).

REMOVAL OF “COMMITMENTS 3” UNIT

This unit began appearing for EU IP addresses performing local searches in 2015 and resembles the 31 January 2014 “Commitments 3” document Google presented to former Commissioner Almunia Its absence in the DMA presentations and documents Google has provided suggests it will no longer be presented.

Analysis1

The effectiveness of Google’s DMA implementation boils down to two key questions:

1) Does the “solution” continue to preference Google’s content above that of rivals? This question is readily answered in the affirmative: Google content continues to be preferenced regardless of quality of content by being displayed alone in the Map, by always being offered on Entity Results Pages, by having access to a Filter Toggle button that uses the general term “Places,” and otherwise. The proposed changes are thus not compliant with DMA 6(5)

2) Yelp also wanted to know if the modification of Google’s SERPs, ostensibly to meet its new legal requirements under the DMA, would even result in a meaningful increase in traffic to third-party services on the world wide web (i e. do the changes increase a user’s propensity to leave Google?).

1 User experience (UX) design is not unlike ballot design As Jonathan Jacobson, one of Google’s top outside antitrust lawyers stated in the context of Google’s search experience, “Consumers will vote with their clicks ” Just as political scientists and pollsters generate predictive data by asking likely voters to respond to hypothetical questions, user experience designers can predict how people behave given a hypothetical search results page with a high degree of accuracy Services such as Lyssna and MTurk allow for rapid, accurate, and inexpensive scaling of this data collection Priming users of these services with intent and offering a plausible hypothetical scenario where they might perform such a search is followed by observation of their click data in a simulated environment. This methodology enables third parties to empirically estimate effectiveness of remedies and hypothetical SERP treatments before they are released to the broader public

In order to answer the second question, we analyzed the effect of the presence of the Google DMA modules compared to a control treatment, examining:

1. The percentage of users who click on a third-party service when they are presented with a control one of Google’s pre-DMA SERPs

2. The percentage of users who click on a third-party service when they are presented with Google’s supposed DMA-compliant SERPs.

3. The percentage of users who, when presented with Google’s new SERPs, click on new features: the VSS results, the “Comparison sites” button in the new SERP Filter Toggles, and the Comparison Sites module.

4. The percentage of users who reach an Entity Results Page who then visit a third-party service via the new links that are provided as part of Google’s DMA compliance proposal

Tests of user behavior when presented with the various SERPs

Yelp ran user tests of various relevant mobile SERP configurations, employing a widely-used service called Lyssna. All tests were meant to emulate a mobile experience, wherein a user was searching for “restaurants in paris” using Google.

TEST 1: A CONTROL SERP, MEANT TO REPLICATE STATUS-QUO SERPS THAT WERE PRESENTED TO EUROPEAN USERS AS OF LATE 2023.

We found that when 250 users were presented with a control SERP, 110 users clicked to the open web — while 133 users clicked in areas that took them to pages operated by Google, or on ads served by Google. An additional 7 failed to follow instructions or clicked on null space. In total, 54 7% of those who followed instructions remained within Google’s walled garden.

TEST 2: A VERSION OF THE PROPOSED SERP THAT GOOGLE CIRCULATED IN DECEMBER — WITH VSS UNIT ABOVE ONEBOX/MAP

We found that when 500 users were presented with this SERP, 210 users stayed within Google’s properties, and 272 moved to or towards the web (We considered clicking on features like the new “Comparison sites” Filter Toggle button to entail movement towards the web, as opposed to the web, because a second click is required to truly exit Google.) The remainder failed to follow instructions or clicked on “null” space.

Only 4 users clicked on the Comparison Sites module, while 11 clicked on the “Comparison sites” Filter Toggle button

In total, 43.6% of those who followed instructions remained within Google’s walled garden.

Full interactive results viewable here: https://app.lyssna.com/tests/b51c2c44fa34/res

TEST 3: A VERSION OF THE PROPOSED SERP THAT GOOGLE CIRCULATED IN DECEMBER — BUT WITHOUT VSS UNIT ABOVE THE ONEBOX/MAP. [THIS IS THE SERP GOOGLE APPEARS TO BE SERVING TO MOST USERS

IN EUROPE.]

We found that when 500 users were presented with this SERP, 354 users stayed within Google’s properties, and 132 moved to or towards the web. The remainder failed to follow instructions or clicked on “null” space. Only 4 users clicked on the Comparison Sites module, while 18 clicked on the “Comparison sites” Filter Toggle button

In total, 72.8% of those who followed instructions remained within Google’s walled garden — a substantial increase relative to the status-quo SERP.

For full results, see: https://app.lyssna.com/tests/3119b320fca6/results/3ea538742d1

Test of user behavior when presented with proposed Entity Results Page module

We then tested the behavior of users who landed on the new Entity Results Page module for a specific restaurant. We found that out of 183 users who were presented with an Entity Results Page for the restaurant Jaja only 2, or 1 1%, engaged with the Yelp or TripAdvisor modules that were presented.

User flow for test of Entity Results Page module

SERP2:VSSBelowOneBox(SERPthatnowusuallyappears)

Conclusion: An Approach that Actually Complies with the DMA

As we have demonstrated, Google’s supposed DMA compliance plans lead substantially more users to remain within Google’s walled garden than did so before the new SERPs were implemented The new Entity Results Page modules generate a negligible amount of traffic to third-party sites. Google continues to exclude all rivals from the Map and OneBox.

Modest tweaks to these proposals might yield a small percentage shift in user behavior but they will not bring Google into compliance with the DMA’s Article 6(5) prohibition on self-preferencing. Compliance will require much more comprehensive reforms, within an entirely distinct paradigm, such as those outlined in the Focus on the User plan.