Google's search quality evaluation process site may have been around for years.

SearchBistro recently posted a 22 page PDF titled General Guidelines on Random-Query Evaluation that was last revised on December 31, 2003. In addition to posting the Random-Query Evaluation PDF, Henk van Ess has recently posted:

• examples of offensive (or low quality) sites

• some whitelisted sites:
  

  Here is a non-exhaustive "white list" of the sites whose pages are not to be rated as Offensive (nor as Erroneous):
  Kelkoo, Shopping.com, dealtime.com, bizrate.com, bizrate.lycos.com, dooyoo.com;

• tips for rating sites:
  

  If it's a machine-generated, no added value affiliates, it's Spam. If it provides some unique values, for example, customer feedback, local information, it should be rated on the merit scale even if it has some affiliates. Similarly, if the game site allows you to download a game, without being intrusive (i.e. install a spyware without notice), it should be rated on the merit scale, instead of Spam.

• How reviewers communicate to come up with solutions when review quality scores are far apart from one another

Search Classification Types:
The Google review guide classifies searches as being

• navigational (example: a search for United Airlines)

• informational (example: how do I..)
• transactional (example: buy 18K White Gold Omega Watch)
• any mixture of the above categories.

Resource Quality Rating:
Google then asks raters to classify sites listed in random queries using the following categories:

• Vital
  • most queries, especially generic type queries do not have a Vital result.

  • Vital result example: search for Ask Jeeves returns www.ask.com.
• Useful
  • these should have some of the following characteristics (although it likely will not exhibit all of them): comprehensive, quality, answers the search query with precission, timly, authoritative.

  • This is the highest rating attainable for most pages on most search queries.
  • Useful result example: search for USA Patriot Act returning the ACLU page covering the USA Patriot Act.
  • For some plural queries, such as Newspapers in Scotland, the best results may be lists of related sites. Reviewers must also check some links on the page to ensure the page is functional.
• Relevant
  • One step down from Useful. Relevant results may satisfy only one important facet of a query, whereas Useful results are expected to be more broad and thorough.

  • Results that would have been Vital if a more common interpretation did not overshadow it are considered relevant.
• Not Relevant
  • Not Relevant results are related to the topic but do not help users.

  • If a person searching for Real Estate finds a San Diego Real Estate website that would probably not be relevant since most people searching for that do not live in or want to move specifically to San Diego.
  • As the San Diego example is too narrow geographically other sites could also be too narrow in other non location based ways, such as being outdated or too specific to a subset idea of the query.
• Off Topic
  • Is not a useful page. Irrelevant.

  • Usually occurs when text matching algorithms do not account for some terms that can have multiple meanings.
• Offensive
  • Pages or sites that often do not hold merit on any query.

  • Example Offensive sites: spyware, unrequested porn, AdSense scraper and other keyword net type sites, etc.
• Erronious
• Didn't Load
• Foreign Language
• Unrated

Vital to Offensive are in order of quality. The higher the better. Erronious through Unrated are cast as non votes. When in doubt between rating values raters are expected to rate at the lower of the two rating values.

Why this is Important:
By learning how and what they want evaluators to look for it makes it easier to understand how to deliver what the search engines want.

This post was a quick review of General Guidelines on Random-Query Evaluation. If you are heavily interested in SEO it is well worth your time to read the original document, which lists many more examples and is in far greater detail than this post.

Random Thoughts:
With how relatively low the wages are for these positions ($10 - $20 an hour) you have to wonder:

• why it took so long for this information to come out

• if some of these people are using the information they gained from participating in other ways
• if these people know anything about Google's business model, and how much THEY could be making on a per click basis if they created well cited content that fit Google's guidelines.
• and a far off tangent! what would happen if Google's business model made self employment too profitable to where they could not afford to pay workers

Not too long ago there were many reports of French trying to rival the Google print program. The International Herald Tribune reports the same thing is now happening in Germany.

Then this year, when Google started wooing publishers to sign on for its own digital book project, that German executive, Matthias Ulmer, decided the time was ripe to seize control with a homegrown counterattack.

Now Ulmer and a five-member task force of the German book trade association Börsenverein are organizing their own digital indexing project, Volltextsuche Online. The effort of the 6,000-member association of booksellers and publishers comes in reaction to Google's plans, unveiled in December, to start digitizing books in the world, with the first step being major university library collections in the United States.

Ultimately variety is going to be important to keep the free flow of information possible. A few companies controlling the information supply is a scary thought, though it looks as though it is where we are headed.

The scalability of search and requested ad networks requires that anyone jumping into the market either

• creates a strong brand in a niche, or

• jumps in big

It will be hard for Google and others to appease publishers as they try to convince them to allow others to control free copies of their content, which at the least will transform the publishing business model and could eventually undermine large segments of it.

Even if some of the uprising forces have little effect on the outcome being a leader in an outraged group helps market the leaders as being market leaders. An article with "challenges Google" in the title is bound to get syndicated thousands of times.

If you can find some angle where you can go against Google which others find nobel it might be some of the cheapest marketing you ever experience.

Whether or not people view Google's desire to control information as evil it is hard to deny that they are at the forefront of pushing others to modernize data.

PageRank:
goes missing from toolbar. Brett Tabke said it is just a temporary glitch though.

Google Engineers:to appear at the New Orleans WMW conference

Does Reciprocal Linking Work?
Recently I saw the Blue Gecko SEO forums ranking at #10 for SEO. Most of his link popularity looked like it was from link trades associated with his webmaster resources directory. The reason people say link trades do not work are mostly because:

• they are usually slow and expensive to build if you do not outsource or automate

• most people exchanging links in bulk are not doing so with quality sites

DMOZ Weighting in Yahoo!:
I created a one page site about Effexor which is listed in DMOZ. I have not built any other linkage data, and it is ranking in the mid 30s for Effexor out of over 7,000,000 sites.

Should write a press release about Google not being about to control their search index, and nefarious webmasters hijacking other sites to remove them from the search index. Why?

Currently the ball is in Google's court, with SEO being branded as spam and scum of the web.

If someone could push the idea that Google could not even control their own index, or how to rate their own site, then perhaps they could somehow shift the frame, saying that Google does not know how to control their index and needs the help of good SEOs to improve their search relevancy.

They should reference:

• Google's general denial of the hijacking problem

• GoogleGuy's comment about how the sites that get screwed are of low quality
• a screenshot of Google's search for AdSense, showing another site ranking #1.

Search engines have yet to be seriously challenged about:

• censoring results without notification, and accepting ads from merchants they would otherwise censor

• freely using webmaster content, and knowingly publishing false or deceptive helpful webmaster information to those same webmasters
• click fraud
• consumer confidence in search relevancy

Most consumers do not realize how search results are manipulated, and most don't even know where the ads are.

It would be cool to see an SEO more daring or less lazy than me use this opportunity to toot their own horn and talk about how they help Google solve a problem it hasn't fully figured out yet - relevancy.

It would certainly be cheap marketing if you get national media coverage with the current feeding frenzy for Google's stock.

Portalized:
Google offers portalization of Google.com. Danny Sullivan has an in depth review. They have a number of features and intend to add many, such as RSS feed support.

Stemming:
Rand points out a post by Xan on stemming and a free online stemming tool

DMOZ:
kills the submission status review. Now its even easier to be corrupt ;)

New York Times:
Begins charging for some of their content. Most of their content remains free. They are also replacing the CEO of About.com.

When Not to Submit to Directories:
when a person creates about a half dozen general directories and promotes them all together. that is not building value, that is trying to cash out and milk the web.

Many directory owners have become exceedingly greedy recently. All the while search algorithms continue to advance and few of the directory owners are actually trying to build any legitimate value.

The Search:
You can pre order John Battelle's new book. He said if you use this link he may be able to autograph it for you, assuming he can work out the shipping details.

The Size of Google's Index:
might have been a bit frothy

Google Factory Tour:
video presentations (should be up soon), Philip Lessen has highlights

Mirago AdSense:
Apparently they have a product similar to AdSense, which might be useful for companies like HotNacho.