What are the major algorithmic differences between the major search engines? I tried answering that question when I recently wrote an article comparing Google to Yahoo! Search to MSN Search.

Please let me know what you think of it.

With Matt Cutts's recent post about the changing quality signals needed to get indexed in Google, and sites with excessive low quality links getting crawled shallower (and some of them not getting crawled at all) some people are comparing Google's current improving crawling standard as an early development of something similar to how the Google Sandbox prevents new or untrusted sites from ranking. WebmasterWorld has a 7ish page thread about BigDaddy, Where Graywolf said:

I'm personally of the opinion that we're starting to see the 'sandbox of crawling'

What is the Optimal Site Size?

Some people in the thread are asking for optimal site size for crawling, or if they should change their internal navigation to accommodate the new Google, but I think to some extent I think that misses the mark.

If you completely changing your site structure away from being usable to do things that might appease Google in a state of flux you are missing the real message they are trying to send. If you rely too heavily on Google then you might find they are in a constant state of being broken, at least from your perspective ;)

The site size should depend largely on

• how much unique content you can create around the topic

• how well you can coax others into wanting to create unique topical content for you
• how people shop
• how people search for information
• how much brand strength you have (a smaller site may make it easier to build a stronger niche specific brand, and in most cases less content of a higher content quality is far more remarkable than lots of junk content)

Many times it is better to have smaller sites so that you can focus the branding messages. When you look at some of the mega sites, like eBay, they are exceptionally weak on deep links, but they also have enough authority, mindshare, and quality link reputation to where they are still represented well in Google.

Scaling Out Link Quality and Unique Content

Another big issue with crawl depth is not only link quality, but also how unique the content is on a per page level. I was recently asked about how much link popularity was needed to index a 100,000,000 page site with cross referenced locations and categories. My response was that I didn't think they could create that much content AND have it unique enough to keep it all indexed AND build enough linkage data to make Google want to index it all.

Sometimes less is more.

The same goes for links too. If you go too hard after acquiring links the sandbox is a real and true phenomenon. If you get real editorial citations and / or go for fewer and higher quality links you will probably end up ranking quicker in Google.

While it may help to be selective with how many links you build (and what sources you are willing to get links from) it also presents a great value to be selective to who you are willing to link at AND link out to many quality resources that would be hard to make look spammy. Rand recently posted

From a trustworthy source - Googlebowling is totally possible, but you need to use patterns that would show that the site has "participated" in the program. What does that mean? Check who they link to - see if you can make the same spammy links point to those places and watch for link spam schemes that aren't in the business of pointing to people who don't pay them.

So if you make your site an island or only partner with other sources that would be easy to take out you limit your stability.

What Makes a Site More Stable?

The big sites that will have 100,000 pages stick in the SERPs are real brands and/or sites that offer added value features. Can individuals create sites to that scale that will still stick? I think they can, but there has to be a comment worthy element to them. They have to find a way to leverage and structure data, be comment worthy and / or they need to have an architecture for social participation / content generation.

The Net Cost & Value of Large Algorithmic Swings

Some people say that in wild search algorithmic swings are not a big deal since that for every person losing someone must gain, so the net effect is not driving people toward paid ads, but I do not buy that.

If your sites are thin spam sites and you have limited real costs the algorithmic swings might not be a big deal, but when businesses grow quickly or have their income sharply drop it affects their profitability, both as they scale up and scale down. You also have to factor in the cost of monitoring site rankings and link building.

At the very least, the ability to turn on or turn off traffic flows (or at least finely adjust them) makes PPC ads an appealing supplement to real businesses with real employees and real business costs. Dan Thies mentioned his liking of PPC ads largely for this reason when I interviewed him about a year ago.

As Google makes it harder to spam and catches spam quicker eventually the opportunity cost of spamming or running cheesy no value add thin sites will exceed the potential profit most people could attain.

Authority Systems Influence the Networks They Measure:

Some people are looking to create systems that measure influence, arguing that as attention grows scarcer it will increase in value:

Attention is what people produce (as in "hand over the money" or "look at this ad") in exchange for information and experience. As Lanham writes in The Economics of Attention, the most successful artists and companies are the ones that grab attention and shape it, in other words, that exercise influence. With so much information, simply paying attention is the equivalent of consuming a meal or a tube of toothpaste.

Any system that measures influence also poses an influence on the market it measures. A retail site in an under-marketed industry that randomly winds up on the Delicious popular list or Memeorandum one day will likely outdistance competitors that do not.

Search has a self reinforcing aspect to it as your links build up. A site with a strong history of top rankings gets links that other sites won't get. Each additional link is a re validation of quality. The people at Google realize that they have a profound effect on how the web grows. Now that they have enough content to establish a baseline in most commercial markets they can be more selective with what they are willing to crawl and rank. And they are cleaning up the noise in their ad market as well.

The Infinite Web, Almost

Some people view the web as an infinite space, but as mentioned above, there is going to be a limit to how much attention and mindshare anything can have.

The Tragedy of the Commons is a must read for anyone who earns a living by spreading messages online, especially if you believe the web to be infinite. While storage and access are approaching free, eventually there is going to be a flood of traditional media content online. When it is easy to link at pages or chapters of books the level of quality needed to compete is going to drastically increase in many markets.

So you can do some of the things that Graywolf suggested to help make your site Google friendly in the short term, but the whole point of these sort of changes at Google are to find and return legitimate useful content. The less your site needs to rely on Google the more Google will be willing to rely on your site.

If you just try to fit where Google is at today expect to get punched in the head at least once a year. If you create things that people are likely to cite or share you should be future friendly.

Caveman recently delivered a terrific post on WMW describing the evolution of SEO, and what it means to be relevant to Google.

...One day, a new search engine named G came along, and decided that if [a man] referred to himself as a pacifist, and others pointed to him as he walked by, then G would rank him as a pacifist.

...It did not take long before the criminal figured out that if the people who pointed to him as he walked by called him pacifist while they pointed, rather than just calling him by his name, his rankings went up for the term "pacifist." So he wore a sign - "pacifist" - and people called him that as they pointed, and his rankings rose.

...After a time, the man realized that if he got all of those he knew to call him pacifist, his rankings would rise further still, and that is what happened.

...So he thought, why not get strangers to call him pacifist, and in return he would refer to them as they wished to be referenced, and all those in his newly expanded network could rank even better for their respective terms. And so it was.

...

...This worked for a while, but eventually, G began to suspect that the faux-pacifists were getting better and better at creating the illusion that they were true pacifists, by begging, borrowing and buying the necessary accolades. It even became known that some faux-pacifists were bribing true pacifists to say nice things about the faux pacifists, so that G would be fooled.

...So, G decided to take drastic measures. They became a registrar so that that could look at each man's historical records. They learned to keep track of what each man said about himself and when, and what others said about each man, and when. And G learned to not trust those who suddenly one day out of the blue proclaimed themselves as pacifists, though their records bore no hint of that previously.

I think this narrative does a terrific job of describing the differences between real and synthetically manufactured authority.

In many small industries there is not much of a topical community, so it may not take much to rank in them, but if there are other legitimate sites ranking for the queries you want to rank for you really have to build reasons why subject matter experts would want to reference you in a positive light.

I think pointing out the social aspect of many links also drives home the concept of a natural editorial citation, and the fact that many real links are driven from social relationships.

Jim Boykin also recently posted on the historical importance of backlinks, and hinted at how he has been cherry picking killer links.

I recently did a linkdomain:seobook.com search on Yahoo! and noticed the search results had "saved by X people" near many of the top listed backlinks.

Results that are bookmarked are typically sites that are frequently visited by real people. As the web becomes more read write the "saved by x people" on the top backlinks will be a great quick SEO tool for testing the depth of support for your site or a competing website.

I consider this website to be somewhat legit, but my sites I would classify as less than legit have no backlinks from pages that are bookmarked on other sites, which is another way of corroborating the value of Jim's philosophy on finding the best pages to get links from within a site.

If you just search for a site it I believe Yahoo! returns those results roughly in order of authority and shows which pages were tagged in those results.

Although more information existing online increases the bar for what is required to be competitive / remarkable / linkworthy I also believe that search engines sharing as much data as they do still makes marketing easy. Thanks to them for that.

The two big downsides to tagging are that it is easy to spam, and currently few people outside of exceptionally techie circles use it much.

I updated my SEO Book PDF today, noting among other things that Matt Cutts sorta hinted that an abundance of low link quality links could cause decreased importance in crawl priority.
Matt then went ahead and posted on his blog that some sites are completely removed from Google's index due to heavy reciprocal linkage.

The sites that fit 'no pages in Bigdaddy' criteria were sites where our algorithms had very low trust in the inlinks or the outlinks of that site. Examples that might cause that include excessive reciprocal links, linking to spammy neighborhoods on the web, or link buying/selling. The Bigdaddy update is independent of our supplemental results, so when Bigdaddy didn't select pages from a site, that would expose more supplemental results for a site.

I think that is probably the post that killed cheesy link exchange networks.

For those who recently said I was full of shit on my position on link exchange networks I am glad that Matt took the time to validate my position. :)

Can other people harm your site? Absolutely. The scalability of the web, and differences between living wages around the world created significant value in funneling around hollow PageRank to sell to naive webmasters which own sites which lack the qualities necessary to be citation worthy.

Knowing that having a certain percentage of shady links will kill your ability to rank in Google adds an additional opportunity cost to building shoddy links which. Things that were once "cheap" or "free" suddenly became expensive, and quality votes gained a bunch more value in the process as well.

This announcement of Matt's in combination with the search results reflecting this activity might be the single biggest thing Google has done in a while to improve the quality of information production across the web.

Fake it till you make it trading to the top still may work well enough in Yahoo! and MSN, but it is not a viable Google solution. If Google could plug some of their other holes they would be much harder to manipulate than most webmasters would like.