I was just fixing up our Robots.txt tutorial today, and figured that I should blog this as well. From Eric Enge's interview of Matt Cutts I created the following chart. Please note that Matt did not say they are more likely to ban you for using rel=nofollow, but they have on multiple occasions stated that they treat issues differently if they think it was an accident done by an ignorant person or a malicious attempt to spam their search engine by a known SEO (in language that is more rosy than what I just wrote).

	Crawled by Googlebot?	Appears in Index?	Consumes PageRank	Risks? Waste?
robots.txt	no	If document is linked to, it may appear URL only, or with data from links or trusted third party data sources like the ODP	yes	People can look at your robots.txt file to see what content you do not want indexed. Many new launches are discovered by people watching for changes in a robots.txt file. Using wildcards incorrectly can be expensive!
robots meta noindex tag	yes	no	yes, but can pass on much of its PageRank by linking to other pages	Links on a noindex page are still crawled by search spiders even if the page does not appear in the search results (unless they are used in conjunction with nofollow on that page). Page using robots meta nofollow (1 row below) in conjunction with noindex do accumulate PageRank, but do not pass it on to other pages.
robots meta nofollow tag	destination page only crawled if linked to from other documents	destination page only appears if linked to from other documents	no, PageRank not passed to destination	If you are pushing significant PageRank into a page and do not allow PageRank to flow out from that page you may waste significant link equity.
link rel=nofollow	destination page only crawled if linked to from other documents	destination page only appears if linked to from other documents	no, PageRank not passed to destination	If you are doing something borderline spammy and are using nofollow on internal links to sculpt PageRank then you look more like an SEO and are more likely to be penalized by a Google engineer for "search spam"

If you want to download the chart as an image here you go http://www.seobook.com/images/robotstxtgrid.png

I recently had an email chat with Alden DoRosario from Chitika about the recent rapid growth of their ad network. They have been aggressively signing up bloggers and other independent publishers, and are now getting over 2 billion monthly impressions, with their behaviorally targeted Premium ads getting hundreds of millions of monthly search driven impressions, putting their search distribution network on par with Ask.com.

How their premium ad network works is they target the ads to be relevant to search query that sent traffic to the publisher's site, thus even if the ads are not shown on a search page they still are seen by searchers right after they search and click through to the site.

Alden gave me a link for a $75 bonus code for any publisher that makes $75 in commissions before the end of October. Publishers are paid 60% of the ad click value, with the house getting 40%. I just added their ads to my mom's weight loss blog. It looks like their ad network is not quite as deep as Google's but they do well for higher volume search queries.

Most search engines are a backbone for an ad network, but it is hard to build query volume for a new search engine. Just look at how few people have used Wikia Search in spite of endless hype. Wikia Search got a couple million lifetime searches whereas Chitika gets billions of monthly ad impressions.

Most people do not feel they have a search problem, but many publishers feel their content could be monetized better. If you didn't have huge search distribution how would you create a search ad network? If an ad network grows big enough do you think they could do it the other way around, using their ad network distribution as a backbone to start a search engine?

Disclaimer: The free $75 bonus is an affiliate link, but when I chatted with Chitika I pushed hard to get publishers the best payout bonus and longest payout bonus period possible rather than focusing on trying to maximize my commissions.

In the past I have mentioned that I am not a fan of doing lots of traditional SEO consulting for a number of reasons (mostly economic), but I still work on a few large projects from time to time. One of the great parts about working with large corporate clients is when you uncover holes in their strategy, finding areas and opportunities that they can own just by deciding to. To some degree it feels like editing the search results, just like a search engineer, seeing you will pushed upon them.

Unlike playing with Wikia Search (which only has a couple millions lifetime searches and nearly a million edits!!!) some of the changes you suggest for enterprise level sites can bring millions of high value visitors to their business free of charge.

When taking on new consulting projects you have to price with the confidence that you will be able to find something that really helps them build their business (and if you are not there is no point taking the project). At first sometimes it can seem like you set the bar too high, but when you do strong research and have a strong partner to bounce ideas off back and forth good things just happen.

Those easy big wins are rare finds, but seem to happen on every project, just in different areas - site structure, duplicate content issues, keyword coverage, internal linking strategy, etc. Digging into a large site with fresh eyes allows you to see things that people who have been close to a project for a long time can not see. Why is that link there? Can this page rank for a couple more related queries? You end up stumbling into something that catches your eye and keep digging in until you have a good solution that earns far in excess of the consulting investment.

With affiliate and AdSense oriented sites the wins are typically much slower, smaller, and harder - sometimes requiring 6 months of effort just to get to break even, and requiring you to fight for every additional link and every additional rank. But the slow and steady path is a stronger business model for SEOs than giving clients millions of dollars of advice for a small fraction of the price. If only I knew how to talk Fortune 500 companies into giving a % of the upside, as that would make consulting so much more profitable than the slow and steady model. :)

Google hires remote quality raters part time for $15 an hour. SEOs working in client based business models usually top out somewhere in the mid 6 figure range. CEOs and some leading web publishers make deep into 7 or 8 figures a year. And some of the early Google engineers might have 9 figures worth of stock. And every one of them is getting paid in part to edit the search results. Does your business model match your ambitions?

I try to teach my mom SEO stuff from time to time, and often do so through the use of analogies. Some analogies perhaps oversimplify the SEO process, but are good for helping get the basic concepts across.

On Page Content

• fish and a fishing pole - when explaining how text heavy sites often outrank thin ecommerce sites, I like to call searchers fish and each word on the page an additional fishing pole in the water. This is really powerful when used in combination with analytics data, showing her the hundreds of phrases that people searched for to find a given page on her site...helping her see the long tail as schools of fish. :)
• Don't Make Me Think - people scan more than they read. Large blocks of text are imposing. People are more likely to read well formatted content that uses many headings, subheadings, and inline links. Expect people to ignore your global navigation, and do whatever you ask them to do (via inline links).

Site Structure

• Broadway Street in Manhattan - used to describe the value of descriptive .com domain names, and when describing what top search engine rankings are worth.
• a pyramid - when explaining how some phrases are more competitive than others, and how to structure a site.
• chapters of a book - used to describe the importance of focused page titles, and how to structure a website.

Link Reputation

• search engines follow people - helps explain why new sites tend to not rank well, and how links are seen as votes.
• roads and highways - used to describe PageRank and why some votes count more than others.
• multiple audiences - used to describe why many types of content are needed to address different audiences, and the importance of creating content that is loved by buyers, linkers, and search engines.
• rising tide lifts all boats - used to describe how links to one part of your website help other pages on your website rank better
• pet rocks & overpriced dolls - describing how perception becomes reality when describing cumulative advantage, and how some poor quality sites are popular while better content remains hidden

Pay Per Click

• instant market feedback - describing how it can be cheaper to test and learn than it is to theorize
• taxing irrelevancy - explaining how irrelevant ads are priced out of the marketplace.
• users vote with clicks - if your ad does not get clicked on it costs way more or is not shown

Tracking Results

• flying blind without autopilot - when explaining the importance of analytics, and how most businesses that do not track results stand a good chance of failing.
• people are lazy - describing the power of defaults and how a #1 ranking gets way more traffic than a number 5 or number 10 ranking.

Google Relevancy

• Trust is Important - cares deeply for user experience, unless they are paid enough to think otherwise.
• The House Advantage - when explaining why YouTube and Knol pages rank better than they deserve to.
• Link Authority is Important - explaining why garbage general made for AdSense sites like eHow clog up the search results when higher quality information is hidden.
• Informational Bias - when explaining that Google's business model relies on people clicking paid ads for commercial sites, and why Wikipedia ranks for everything.

How do you describe SEO to people who are not deep into the field?

Update: A few months ago Jaan Kanellis posted many analogies.

cNet recently covered a new Microsoft Search research paper on BrowseRank [PDF]. The theory behind the concept of BrowseRank is that rather than using links (PageRank) as the backbone of a relevancy algorithm, you could look at actual usage data from hundreds of millions of users.

Since there are more web users than webmasters BrowseRank would be a more democratic system, but many users are mislead and/or easily influenced by social media, public relations, and some referral spam strategies, so BrowseRank could surface some low quality temporal information, making manipulating Digg and other "firehose of traffic" sources more valuable than they perhaps should be. Although if certain referrals were blocked (Digg, StumbleUpon, etc.) and/or BrowseRank was combined with a blended search strategy (like how Google mixes Google News in their organic results) Microsoft could have a bit more confidence in waiting out some traffic spikes to see if traffic is sustained. And this potential shortfall (if managed properly) could actually lead to a major advantage over the stale effect of PageRank. If you create non-resource hyped up piece of linkbait that gets a quick rush of links and never gains any more votes then why should that page have a lot of authority to pass around your site?

BrowseRank can look at user time on site to help determine if the site was of quality, and perhaps even normalize for page length, but what happens if a page is really good at answering a common question? Even if people only ask it once in a great while quality content should not be penalized for great formatting, ease of use, and a great user experience - though as search evolves search engines will keep displaying more content in the search results, license specialized data, and answer many common questions directly in the search results.

In addition to looking at traffic that comes via links, BrowseRank also identifies direct URL visits via bookmarks or typing in URLs. These types of traffic sources

are considered "green traffic" because the pages visited in this way are safe, interesting, and/or important for users.

Such an algorithm would add value to direct navigation keyword rich URLs. Another obvious extension of such an algorithm would be identifying brand specific searches and URL searches, and bucketing those referrals into the green category as well.

To encourage such branded search queries and long user interactions it would be better to create strong communities with repeat visitors and many web based tools rather than allowing useful user interactions occur through browser extensions.

Another big issue with BrowseRank is that it highlights many social media sites. The issue with social media is that any piece of content is generally only relevant to a small number of people and most of the content is irrelevant to the population at large. Unless the search engine had a lot of personalized data promoting the general purpose social media sites would be blunderous - surfacing lots of results that are irrelevant, spam, or both.

One of the big advantages PageRank has over BrowseRank is an economic one.

• People are more likely to link at informational resources, thus surfacing those pages and sites higher in the search results.
• This gives Google's organic search results an informational bias which makes searchers more likely to click on Google's paid ads when performing a commercial search.
• Google also has the ability to arbitrarily police links and/or strip PageRank scores to 0 with the intent to fearmonger and add opportunity cost to anyone who gathers enough links pointing at a (non-corporate owned) commercial domain. This layer of social engineering coerces publishers to create the type of content Google likes to rank.