How To Measure Bias In Google's Results

Here's an interesting study, conducted by Benjamin Edelman and Benjamin Lockwood, from the Harvard Business School. The study measures how much search engines, Google in particular, favor their own web services.

We find that each search engine favors its own services in that each search engine links to its own services more often than other search engines do so. But some search engines promote their own services significantly more than others.... we find that Google's algorithmic search results link to Google's own services more than three times as often as other search engines link to Google's services. For selected keywords, biased results advance search engines' interests at users' expense

People have debated this topic for a while, some saying the search engines can do what they like, others feel the search engines must be held to account.

However, the study brings up an important point. If Google claims to have algorithmic, "objective" search results, then it follows that Google should not favor their own companies properties, unless those properties achieve a top ranking based on their own merit.

Google can't have it both ways.

The problem, of course, is that Google could tweak the algorithm to favor whatever qualities its own properties display e.g. the PageRank of Google's own pages could be calculated - in truly cryptic and oblique fashion - as being of higher "worth". After all, there's no such thing as "objective" when it comes to editorial, which is the function of a search algorithm. There are merely points along a continuum of subjectivity.

But where it gets interesting is the study goes one step further. It tries to figure out what the user wanted when she searched. Did the user want to find a Google service at #1? And if not, then isn't Google doing the user a dis-service by placing a Google property at #1?

In principle, a search engine might feature its own services because its users prefer these links. For example, if users of Google's search service tend to click on algorithmic links to other Google services, whereas users of Yahoo search tend to click algorithmic links to Yahoo services, then each search engine's optimization systems might come to favor their respective affiliated services. We call this the "user preference" hypothesis, as distinguished from the "bias" theory set out above

They tested this theory using click-thru data. Regarldess of the search keyword, users almost always favor the #1 result - 72% of the time. So what if the user clicks further down, indicating that the first result is less relevant?

Gmail, the first result, receives 29% of users' clicks, while Yahoo mail, the second result, receives 54%. Across the keywords we looked at, the top-most result usually receives 5.5 times as many clicks as the second result, yet here Gmail obtains only 53% as many as Yahoo. Nor was "email" the only such term where we found Google favoring its own service; other terms, such as "mail", exhibit a similar inversion for individual days in our data set, though "email" is the only term for which the difference is large and stable across the entire period

There is a huge incentive for search engines, which increasingly crossing the line into publishing territory, to skewer the results towards their own properties. The traffic is valuable, and, whatismore, can be channeled away from competitors.

As Aaron pointed out a few months ago, if Google choose to enter a new vertical, such as travel or local, then you'd better watch out if you compete in those verticals. Regardless of how relevant you are to the search term, it's below-the-fold you'll likely be going.

So, yes, it may be Google's search engine, but they can't make claims about focusing on the user above all else, otherwise they'd return results the user wants, as opposed to possibly directing the user to Google properties due to other considerations. How can they claim "Democracy works", if they don't favour whatever site the link graph "votes" most relevant? And doesn't this come down slightly on the wrong side of "evil"?

So, What To Do?

If you feel Google can position their own sites where they like, then nothing.

Personally, I think any company can do what they like, until they reach a point where they become so influential, they can use their sheer size to reduce competition and choice. If we believe that free markets require healthy competition in order to thrive, then we should be wary of any entity that can reduce competition using anti-competitive behavior.

I'm not saying that is what Google is doing, but watch this space. Some European agencies are investing allegations of anti-trust violations.

The Commission will investigate whether Google has abused a dominant market position in online search by allegedly lowering the ranking of unpaid search results of competing services which are specialised in providing users with specific online content such as price comparisons (so-called vertical search services) and by according preferential placement to the results of its own vertical search services in order to shut out competing services

The fact Marissa Mayer said this:

[When] we roll[ed] out Google Finance, we did put the Google link first. It seems only fair right, we do all the work for the search page and all these other things, so we do put it first... That has actually been our policy, since then, because of Finance. So for Google Maps again, it’s the first link

....makes matters......interesting ;)

Secondly, if you're big enough, you could make a point of taking Google on. Check out Trip Advisors take on Google Places displaying Trip Advisors data in repackaged form, which could cause Google users to stay on Google, and not go to the Trip Advisor site:

Google is no longer able to stream in reviews from TripAdvisor to Places pages after the user review giant blocked it. TripAdvisor confirmed the move today in an email, stating that while it continues to evaluate recent changes to Google Places it believes the user does not benefit with the “experience of selecting the right hotel”. As a result, we have currently limited TripAdvisor content available on those pages,” an official says

But Google aren't really going to care much about you if you don't have some major clout.

Thirdly, stay out of any vertical Google is likely to want to own. It is likely that Google will be going after the big verticals, because a big company needs to score big on projects. Long tail stuff isn't going to make any difference to their bank balance, except in aggregate, so there will be millions of verticals in which you'll never face a direct threat.

This is also a timely reminder to build up your non-search traffic in case Google, or any other search engine, decides to change the game significantly in their favor. Encourage users to bookmark, develop your social media brand, build mailing lists, put some valuable content behind log-in/pay walls, and build membership sites. Relying on Google has always been a risky strategy, do diversify your traffic strategy where you can in 2011.