[Update: use this supplemental ratio calculator. Google is selfish and greedy with their data, and broke ALL of the below listed methods because they wanted to make it hard for you to figur out what pages of your site they don't care for. ]

A person by the nickname DigitalAngle left the following tip in a recent comment

If you want to view ONLY your supplemental results you can use this command site:www.yoursite.com *** -sljktf

Why Are Supplemental Results Important?

Pages that are in the supplemental index are placed there because they are trusted less. Since they are crawled less frequently and have less resources diverted toward them, it makes sense that Google does not typically rank these pages as high as pages in the regular search index.

Just how cache date can be used to view the relative health of a page or site, the percent of the site stuck in supplemental results and the types of pages stuck in supplemental results can tell you a lot about information architecture related issues and link equity related issues.

Calculate Your Supplemental Index Ratio:

To get your percentage of supplemental results you would divide your number of supplemental results by your total results count

site:www.yoursite.com *** -sljktf
site:www.yoursite.com

What Does My Supplemental Ratio Mean?

The size of the supplemental index and the pages included in it change as the web grows and Google changes their crawling priorities. It is a moving target, but one that still gives you a clue to the current relative health of your site.

If none of your pages are supplemental then likely you have good information architecture, and can put up many more profitable pages for your given link equity. If some of your pages are supplemental that might be fine as long as those are pages that duplicate other content and/or are generally of lower importance. If many of your key pages are supplemental you may need to look at improving your internal site architecture and/or marketing your site to improve your link equity.

Comparing the size of your site and your supplemental ratio to similar sites in your industry may give you a good grasp on the upside potential of fixing common information architecture related issues on your site, what sites are wasting significant potential, and how much more competitive your marketplace may get if competitors fix their sites.

WebmasterWorld has been running a series of threads about various penalties and filters aligned with specific URLs, keyword phrases, and in some cases maybe even entire directories.

Some Threads:

There is a lot of noise in those threads, but you can put some pieces together from them. One of the best comments is from Joe Sinkwitz:

1. Phrase-based penalties & URL-based penalties; I'm seeing both.
2. On phrase-based penalties, I can look at the allinanchor: for the that KW phrase, find several *.blogspot.com sites, run a copyscape on the site with the phrase-based penalty, and will see these same *.blogspot.com sites listed...scraping my and some of my competitors' content.
3. On URL-based penalties allinanchor: is useless because it seems to practically dump the entire site down to the dregs of the SERPs. Copyscape will still show a large amount of *.blogspot.com scraping though.

Joe has a similar post on his blog, and I covered a similar situation on September 1st of last year in Rotating Page Titles for Anchor Text Variation.

You see a lot more of the auto-gen spam in competitive verticals, and having a few sites that compete for those types of queries helps you see the new penalties, filters, and re-ranked results as they are rolled in.

Google Patents:

Google filed a patent application for Agent Rank, which is aimed at allowing them to associate portions of page content, site content, and cross-site content with individuals of varying degrees of trust. I doubt they have used this much yet, but the fact that they are even considering such a thing should indicate that many other types of penalties, filters, and re-ranking algorithms are already at play.

Some Google patents related to phrases, as pointed out by thegypsy here:

• Phrase-based searching in an information retrieval system

• Multiple index based information retrieval system
• Phrase-based generation of document descriptions
• Phrase identification in an information retrieval system
• Detecting spam documents in a phrase based information retrieval system

Bill Slawski has a great overview post touching on these patent applications.

Phrase Based Penalties:

Many types of automated and other low quality content creation cause the low quality pages to barely be semantically related to the local language, while other types of spam generation cause low quality pages to be too heavily aligned to the local language. Real content tends to fall within a range of semantic coverage.

Cheap or automated content typically tends to look unnatural, especially when you move beyond comparing words to looking at related phrases.

If a document is too far off in either direction (not enough OR too many related phrases) it could be deemed as not relevant enough to rank, or a potential spam page. Once a document is flagged for one term it could also be flagged for other related terms. If enough pages from a site are flagged a section of the site or a whole site can be flagged for manual review.

URL and Directory Based Penalties:

Would it make sense to prevent a spam page on a good domain for ranking for anything? Would it make sense for some penalties to be directory wide? Absolutely. Many types of cross site scripting errors and authority domain abuses (think rented advertisement folder or other ways to gain access to a trusted site) occur at a directory or subdomain level, and have a common URL footprint. And cheaply produced content also tends to have section wide footprints where only a few words are changed in the page titles across an entire section of a site.

I recently saw an exploit on the W3C. Many other types of automated templated spam leave directory wide footprints, and as Google places more weight on authoritative domains they need to get better at filtering out abuse of that authority. Google would love to be able to penalize things in a specific subdomain or folder without having to nuke that entire domain, so in some cases they probably do, and these filters or penalties probably effect both new domains and more established authoritative domains.

How do You Know When You are Hit?

If you had a page which typically ranked well for a competitive keyword phrase, and you saw that page drop like a rock you might have a problem. Other indications of problems are if you have inferior pages that are ranking where your more authoritative page ranked in the past. For example, lets say you have a single mother home loan page ranking for a query where your home loan page ranked, but no longer does.

Textual Community:

Just like link profiles create communities, so does the type and variety of text on a page.

Search results tend to sample from a variety of interests. With any search query there are assumed common ideas that may be answered by a Google OneBox, related phrase suggestions, or answered based on the mixture of the types of sites shown in the organic search results. For example:

• how do I _____

• where do I buy a ____
• what is like a _____
• what is the history of ______
• consumer warnings about ____
• ______ reviews
• ______ news
• can I build a ___
• etc etc etc

TheWhippinpost had a brilliant comment in a WMW thread:

• The proximity, ie... the "distance", between each of those technical words, are most likely to be far closer together on the merchants page too (think product specification lists etc...).

• Tutorial pages will have a higher incidence of "how" and "why" types of words and phrases.
• Reviews will have more qualitative and experiential types of words ('... I found this to be robust and durable and was pleasantly surprised...').
• Sales pages similarly have their own (obvious) characteristics.
• Mass-generated spammy pages that rely on scraping and mashing-up content to avoid dupe filters whilst seeding in the all-important link-text (with "buy" words) etc... should, in theory, stand-out amongst the above, since the spam will likely draw from a mixture of all the above, in the wrong proportions.

Don't forget that Google Base recently changed to require certain fields so they can help further standardize that commercial language the same way they standardized search ads to have 95 characters. Google is also scanning millions of books to learn more about how we use language in different fields.

Ever since Google has got more selective with what they will index, the model for profitable SEO changed from chucking up pages and hoping some of them are profitable, to where it makes sense to put more strategy into what you are willing to publish.

The Supplemental Index Hates Parasitic SEO:

Each site will only get so many pages indexed given a certain link authority. And each of those pages will rank based on the domain's authority score, and the authority of the individual page, but each page needs a minimum authority score to get indexed and stay out of the supplemental results - this is how Google is trying to fight off parasitic SEO.

Given that many people are leveraging trusted domains, it makes sense that if you have one that you leverage it in a way that makes sense. CNN will rank for a lot of queries, but it does not make sense for Google to return nothing but CNN. It is good for the health of Google to have some variety in their search results. This is why smaller sites can still compete with the bigger ones, Google needs to use the smaller sites to have variety and to have leverage over the larger sites...to keep the larger sites honest if they are too aggressive in leveraging their authority, or have holes that others are exploiting.

Extending a Profitable Website:

If you have a 100 page niche website you may be able to expand it out to 500 pages without seeing too much of a drop in revenue on those first 100 pages, but eventually you will see some drop off where the cost of additional content (via link authority that it pulls from other pages on your site) nearly matches the revenue potential of the new pages. And then at some point, especially if you are not doing good keyword research, have bad information architecture, create pages that compete with other pages on your site, are not actively participating in your market (gaining links and mindshare), or if you are expanding from a higher margin keyword set to a lower margin one, you may see revenues drop as you add more pages.

The solution to fix this problem is build editorial linkage data and stop adding pages unless they have a net positive profit potential.

What are the costs of content?

• the time and money that went into creating it

• link equity (and the potential to be indexed) that the page takes from other pages
• the mindshare and effort that could have been used doing something potentially more productive
• the time it takes to maintain the content
• if it is bad or off topic content, anything that causes people to unsubscribe, hurts conversion rates, or lowers your perceived value is a cost

How can a Page Create Profit?

• anything that leads people toward telling others about you (links or other word of mouth marketing) is a form of profit

• anything that makes more people pay attention to you or boosts the credibility of your site is a form of profit
• anything that thickens your margins, increases conversion rates, or increases lifetime value of a customer creates profit
• anything that reduces the amount of bad customers you have to deal with is a form of profit

Mixing Up Quality for Profit Potential:

I am still a firm believer in creating content of various quality levels and cost levels, using the authoritative content to get the lower quality content indexed, and using the lower quality content earnings to finance the higher quality ideas, but rather than thinking of each page as another chance to profit it helps to weigh the risks and rewards when mapping out a site and site structure.

Increasing Profit:

Rather than covering many fields broadly consider going deeper into the most profitable areas by

• creating more pages in the expensive niches

• making articles about the most profitable topics semantically correct with lots of variation and rich unique content
• highly representing the most valuable content in your navigational scheme and internal link structure
• creating self reinforcing authority pages in the most profitable verticals
• requesting visitors add content to the most valuable sections or give you feedback on what content ideas they would like to see covered in your most valuable sections
• If your site has more authority than you know what to do with consider adding a user generated content area to your site

Take Out the Trash:

If Google is only indexing a portion of your site make sure you make it easy for them to index your most important content. If you have an under-performing section on your site consider:

• deweighting it's integration in the site's navigational scheme and link structure

• placing more internal and external link weight on the higher performing sections
• if it does not have much profit potential and nobody is linking at it you may want to temporarily block Googlebot from indexing that section using robots.txt, or remove the weak content until you have more link authority and/or a better way to monetize it

If you put one form of internal navigation in parallel with another you are essentially telling search engines that both paths and both subset pages are of the same significance. Many websites likely lose 20% or more of their potential traffic due to sloppy information architecture that does not consider search engines.

Many people believe that having more pages is always better, but ever since Google got more aggressive with duplicate content filters and started using minimum PageRank thresholds to set index inclusion priorities that couldn't be further from the truth. Shoemoney increased his Google search traffic 1400% this past month by PREVENTING some of his pages from being indexed. Some types of filtering are good for humans while being wasteful for search engines. For example, some people may like to sort through products by price levels or look at different sizes and colors, but pages that are almost duplicate with the exception of price point, size, model number, or item color may create near duplicate content that search engines do not want to index.

If you are wasting link equity getting low value noisy pages indexed then your high value pages will not rank as well as they could because you wasted link equity getting low value pages indexed. In some cases getting many noisy navigational pages indexed could put your site on a reduced crawling status (shallow crawl or less frequent crawl) that may preclude some of your higher value long tail brand specific pages from getting indexed.

More commonly searches that have some sort of filter associated with them will be associated specific brands rather than how we sort through those brands via price-points. Plus the ads for those terms tend to be more expensive as well.

The reasons brands exist is that they are points of differentiation that allow us to charge non commodity prices for commodities. That associated profit margin and marketing driven demand is why there is typically so much more money in branded terms than other non-brand related filters.

When designing your site's internal link structure make sure that you are not placing noisy low value pages and paths in parallel or above higher value paths and pages.

Google claimed they killed off the ability to GoogleBomb sites. Matt Cutts stated that it should not affect people SEOing their own sites, but some SEOs are already talking about GoogleBombing competitors to knock them out of the search results.