Indexed Page Quality Ratio

Hehe... that's one of the tricks I've discovered this year. On my more important sites I block the SEs from indexing anything that isn't THE EXACT PAGE that I want people to find.

While some might worry about not having as many pages as possible indexed, my traffic has only gone up since implementing this.

It takes some brain work, but by using robots.txt, and also Meta Tags, you can pretty much get Google to only index the good stuff.

In fact my main site has a perfect ratio of indexed pages - zero filler, 100% tasty :)

The main functional portion of my site is dynamic so that I can allow the content archives to be updated by only changing my XML file. This is user generated content based on the month that the user wants to view. Because of my concern for the search engines indexing dynamic content, I have created mirror pages made up of static html of the months that you can get to by the sitemap. Obviously I can only do this when a month is over.

Is this considered duplicate content in a bad way? Does the search engine index my XML file as well and look at this content as 3 times in repetition?

Cary - that sounds interesting - I use html sitemaps to ensure most of my site is indexed, however I find many visitors arrive on those sitemap pages rather than the pages with content they are looking for, and then leave.

Is there any way of ensuring the pages I'm pointing to are indexed by google without having these sitemap pages in the index?

How would you recommend handling spidering for something like a blog, where the content on the site index changes frequently, and search engine listings often include content that is no longer on that page? People end up clicking through to the site index instead of the permalink page and being unable to find the content they are looking for.

But not letting engines index your site index would be a really bad idea, wouldn't it?

I noticed that one of my main competitors that has a very large market share and a high pagerank has included the Meta "no index, no follow" on their main landing page. Why is that a good idea and how is that successful?

Rand,

If your reading this post.

This might be an interesting item to include on your page strength tool.

James: You can always use meta tags (I assume you're not talking of Google Sitemaps) setting index to "noindex, follow"

This way Google will follow all links of that page but it won't index the page itself.

Nancy: Good question. I'd like to get more info on this topic, too. If you have found out anything, please let me know!

Hi Blake
Not sure if you need to make a mirror html file if you already have a php one or something like that. Many sites have feeds and XML files, so that probably shouldn't be too big of a problem since your link structure will most likely reinforce your home page or main blog page as being more authoritative than your feed, and many feeds are not full text feeds in nature (using only snippets).

One note of caution with feeds is that you want to make sure you host your feed on your own site. If you host it on an external more authoritative domain that can cause subscribe transfer problems if you ever leave them AND I have seen some search engines rank those feeds on more authoritative domains (like Feedburner) above the original domain the content came from.

Hi James
You don't need sitemaps if your internal site structure has clean logical hierarchical links and clean URLs. Sitemaps are nice for giving you another means to get indexed, a way to focus / alter / change internal anchor text, and a way to redistribute your PageRank / link authority, but if you have a small site that is manually crafted or a rather advanced content management system you should be able to integrate some of those ideas directly into your site without using a sitemap.

Hi Nancy
I think if your site gets enough authority (via legitimate links) if the search engine is good then eventually they should get better at indexing it and sending people to the correct page.

I definently would want to get my main page indexed, but you also want to ensure that your sub-pages have clean URLs and are only getting indexed at one location.

Hi Blake
I can't really state why anyone would use a technique and be certain, plus their use of a technique may help or hinder them.

Aaron/Others: Can you elaborate on bullet point # 4 above, namely "dangling nodes that act as PageRank sinks." I'm wrestling with a Drupal site at the moment that creates all kinds of nodes. Wondering if you've unravelled a part of the mystery?

A dangling node is a page which has inbound links, but does not filter the link authority back to other pages within the site.

Cary, That sounds interesting to me but it seems like a lot of trouble. Were you having issues with the se sendign users tot he wrong pages who then balked and moved on? I always sort of figured that I'd make content as best as I could and hope that the search engine sent the right people to the right places. Otherwise it seems like I'd only be getting traffic for one term or concept vs. the many that a whole site typically contains. That may just be fancy talk though to mask my laziness. Either way, kudos.

Aaron, I appreciate the def for dangling nodes. I'd never really thought about it before.