SEO & PPC Competitive Analysis & Keyword Research Tools

Background Information on this Post:
There are getting to be a ton of keyword tools on the market, so I decided to test most any keyword tool I have heard of.

To get a full in depth understanding of all of these keyword tools you need to run them on a variety of terms. Having said that, I am going to compare how well they track search volume to a single niche long running AdWords ad group.

One of my clients is a distributor for a specific brand of products.
The company name is something like ABC Tires

My AdWords keyword group containing phrases like

ABC Tires
A B C Tire
A B C Tires retailer
ABC Car tires
ABC Tire co
ABC Tires company
ABC Tires.com
ABC Wheels
etc

I have had over 100,000 ad displays in the last year and a half. Of about 60 keywords in that keyword group Google has sent my client traffic for about 30 of them, and 11 of them have converted to sales. One of the terms that converted to a sale was ABC, which had the lowest conversion cost, but was quickly disabled since the clickthrough rate was too low.

I am using that Google search ad distribution as the baseline for comparing the following tools. My AdWords ad group is targeted at US only. I do realize there is bias in only taking data from one region from one engine and only looking at one keyword set, but the data is collected from the largest engine in the largest market.

Since this manufacturer is rather niche and somewhat unheard of on the web it presents a good opportunity to see how well these tools do deep keyword research.

Google Keyword Tool
try Google's Keyword Tool free

Benefits:

• shows 12 months historical search trends

• can analyze a page or site for automatically recommending keywords
• they have the largest search userbase, and thus can offer great keyword depth
• gives estimated cost, competition level, and suggests potential negative keywords
• easy to export your list as an XLS spreadsheet

Negatives:

• this tool is well integrated into the AdWords system, so if your competitors have half a clue they should be bidding on these terms

• search engines may desire to show you terms which offer the highest prices instead of the highest search volumes
• the estimates are exceptionally rough

Google Suggest:
try Google Suggest free or try our Google Suggest scraper free

Benefits:

• Free, quick, & easy to use.

• Google has a huge database of search activity to grab this data from.
• Google's Toolbar auto updates and they added Google Suggest as a function directly to the toolbar in Google Toolbar 4 beta. They also have created a FireFox Google Suggest extension.
• Google updating their toolbar and potentially their main site to add this feature could cause the terms listed to increase in search volume

Negatives:

• It only shows up to 10 terms for any search term and does not give any sort of search volume estimate.

• Only shows terms that specifically start with the letter sequence you type it. This will not help you find related searches that have modifiers before the core term. On a positive note this does make it easy to see what are the most common ending modifiers for plural and singular versions of search terms.
• Like most Google products they are not particularly clear how this data is organized.

ABC Tires:
For my ABC Tires example Google Suggest recommended 10 different terms, (and a few more for A B C tires) and about 75% of their suggestions led to conversions.

Google Keyword Sandbox:
try Google Keyword Sandbox free

Benefits:

• Free, quick, & easy to use.

• In addition to showing exact search terms it also shows many related search terms and potential modifiers.
• Google is the largest search engine and has access to more search data than any other company. (They also bought Urchin web analytics, so they can even track some data from other engines.)

Negatives:

• Since Google is the largest engine many people may use this tool and bid up these terms.

• Like all things Google, they do not share specifics on how they gather this data or what it means to you.

ABC Tires:
For my ABC Tires example Google Keyword Sandbox recommended 4 different terms, and all of them have converted for me. It also listed a wide variety of semantically related search terms and modifiers, which can be used to help extend out a keyword list.

Google also has a highly useful keyword research tool within their AdWords interface when you log in. It automates keyword research based on entering a URL or site. It also allows you to find related keywords based on words you enter or words that are already in your account. The biggest downsides to the Google tools are that their search volume estimates suck and you can be fairly certain that some of your competitors will also be using the tools built into the AdWords system. Based on using all of these tools I believe that combining that tool with the SEO Book keyword research tool is just about all you need to use.

Overture (which is being rebranded as Yahoo! Search Marketing) Keyword Suggestion Tool:
try the Overture Keyword Suggestion Tool free

Benefits:

• It is quick, free, and easy to use.

• Overture has a large partner network to gather data from.
• Overture also offers a view bid price tool
• Overture offers search research by geographic region

Negatives:

• Many bid management tools and rank checkers scour the Yahoo! network, causing traffic numbers to be heavily skewed toward high value search terms.

• Overture does not seperate out plural and singular versions of keywords.
• Does not show related terms, only terms containing the exact phrase you search for.
• This was the original free search term suggestion tool, so everyone and their dog who does keyword research looks at this data.

ABC Tires:
For my ABC Tires example Overture only showed the four most common terms, all of which lead to many conversions, but were some of the most expensive terms.

SEO Book Keyword Research Tool:
try the SEO Book Keyword Research Tool for free

Benefits:

• Free.

• Easy to switch between markets.
• The results cross references many keyword research tools (almost every tool listed on this page - and a few that are not) so you can mix and match using them quickly and easily.
• Also links to different information systems (news search, directories, blog search, tagging sites, etc.) to help you find useful resources and things people are discussing in your marketplace.

Negatives:

• Driven off of Overture, so any problems with Overture's data occur in my tool as well.

WordTracker:
subscribe to WordTracker

Benefits:

• Helps you find or think of related terms by offering a built in thesaurus and lateral search. The lateral search looks at page details such as keywords of pages which are thought to be competing or related to your topic.

• WordTracker allows API keyword research access.
• Has more features than most other keyword research tools.
• The added features, such as showing the lateral terms, saves me a ton of time by showing me many related terms and useful modifiers I could or should be using.
• In the grand scheme of things the cost is not that expensive. It also has flexible pricing which allows you to subscribe for about $8 for a day or $250 for a year.
• Since it grabs it's data from Dogpile and MetaCrawler (a couple meta search engines) it's data is usually more clean than data from systems which collect data from networks that rank checkers and bid management tools work on.
• WordTracker offers a free trial, which the free Digital Point Keyword Suggestion tool queries.

Negatives:

• It is easy for competitors to spam the WordTracker database.

• Only grabs data from a couple meta search engines.
• Last time I checked it they had 305,005,525 queries in their database from 95 days, which is probably less queries than Google serves in a day.
• I do not think WordTracker sorts information into regional databases.
• Since they only gather data from a small sector of the web their data sampeling errors are magnified. Sometimes they will make low volume terms seem more important than they are, and other times some terms will not show up.

ABC Tires:
For my ABC Tires example they offered a ton of modifier words and about a dozen search terms (including the modifiers I was able to make many additional keywords). A couple of the terms they showed me were obvious random one off type searches that a random surfer searched for twice, but many of the terms lead to conversions. The lateral search and thesaurus make WordTracker great.

Digital Point Keyword Research Tool:
try Digital Point's Free Keyword Research Tool free

Digital Point's keyword tool compares Overture and WordTracker data side by side.

Benefits:

• It is quick, free, & easy to use.

• It gives multiple suggestions side by side.
• It is the sum of the pieces of WordTracker and Overture's Search Term Suggestion Tool.

Negatives:

• It lacks some of WordTrackers more advanced features.

Keyword Intelligence:
subscribe to Keyword Intelligence ($89 to $189 per geographic market)

Benefits:

• It costs ($89 to $189 per market), so not everyone has access to the data.

• Offers 3 months historical data.
• They claim to track 25 million actual web users. Most marketers do not have direct access to spamming this data.
• Shows keyword research data by geographic market.
• Tracks actual traffic.
• Tracks keyword success rates, which shows what percent of clicks ended up in people clicking a result.
• Breaks results down by market.
• They made the industry data a downloadable spreadsheet & the industry data keywords allow you to click on them to pull up the deeper terms containing that search term.
• Viewing the success rates for a good number of terms gives you access to how people search and helps you better understand the psychology of search.

Negatives:

• Considering that Google.com has hundreds of millions of searches daily, the market data from only 25 million web users is a bit limited.

• It has a monthly cost of $89 to $189 per market.
• It offers industry specific keyword terms, but in most cases I found many of the terms overtly generic, and they need to offer additional filtering options from within those industries to make the data more useful. Plus they only list 100 terms per page, which makes some of data not as easy to work with.
• Some of the industry terms are mistargeted. Foo Fighters is not a web development term.
• I did not notice a related terms function. Surely with the data they have access to and some of the company they have partnerships with they could help recommend related terms, and not just terms having the exact words you enter in them. If you are just giving me terms that are extensions of what I am typing in then why don't I just grab more terms by running a Google AdWords account and tracking the keyword data in referal strings?

ABC Tires:
For my ABC Tires example Keyword Intelligence only showed me 3 term variations. All 3 variations were terms that converted, but only giving me three of the most broad variations does not allow me to get cheaper and hypertargeted clicks. In the end that still leaves me focused on the most broad and overpriced terms.

I am still a bit new to the Keyword Intelligence tool, but Mikkel deMib Svendsen recently posted a similar view on ThreadWatch:

My problem with the HitWise data has always been that it seems to only cover the "top" of the keyword pile. As an example, I had a seles rep from HitWise run a report for a client I work for in a very competitive market. The report showed that the competitors got more search traffic than my client but not one single of the approx 35-40,000 keyword my client gets their 300k monthly search visitors from was listed!

So actually I was quite happy - at least if our competitors buy this because they look at it and think: We are doing pretty good, when in fact they are not :)

Random:
When I searched for my own name Keyword Intelligence did bring up that people are searching for my name and military discharge status and code (searches like oth navy re-4 aaron wall). Sorta feels weird knowing that I can see that other people are researching my past. Overture, WordTracker, etc did not show that particular search.

I have not tried HitWise's competitive intelligence stuff yet (where you see what search terms and websites drive traffic at competing sites) but it might have value for large corporate accounts. I believe HitWise starts out at about $10,000's / year. With that kind of price I would have thought that their keyword research segment within the HitWise service (I believe it heavily overlaps with Keyword Intelligence) would have many more features than it does.

Gavin Appel stated Keyword Intelligence includes a limited sub set of data and search term analysis features that are available within the Hitwise Competitive Intelligence service and sent this comparison URL: http://www.keywordintelligence.com/differences.php

Keyword Discovery:
subscribe to Keyword Discovery ($49.95 per month)

Benefits:

• Since it costs not everyone has access to the Keyword Discovery research database. They do not state which engines they have partnerships with. That combined with the limited exposure means that Keyword Discovery is less likely to be spammed than other SEO related keyword research products.

• If you buy the $300 per month version Keyword Discovery allows API keyword research access.
• Shows keyword search trend throughout the year.
• Has many interesting syntaxes and features, allowing you to:
  • compare a keyword list to a URL and find what terms occur in that page

  • allows you to search for common spelling errors associated with your keyword
  • subtract out certain terms from your search query

Negatives:

• Since they do not state where their data comes from it is hard to know exactly how pure it is. If a bunch of it comes from affilate traffic from small pay per click search providers then the traffic will be naturally biased toward the most expensive and frequently searched terms.

• I still like WordTracker a bit better, but part of that might be that I am more used to using WordTracker.

ABC Tires:
For my ABC Tires example they had 6 examples, 4 of which converted. I believe their database is larger than WordTracker's, but I think at the 2005 NYC SES conference Dan Thies stated that WordTracker's database is typically cleaner data.

Keyword Discovery also has a free keyword directory which shows some of the terms that drive traffic at DMOZ category listed websites, and allows access to the top 10 results free via search. Since their data seems to be a bit top heavy you might be able to get all the useful info by just glancing at the free trial information.

MSN Submit It / B Central Keyword Research Tool:
no try now link, reasons explained below...

Benefits:

• not everyone has access to it

• shows related terms, for example seo and search engine optimization return things like internet marketing, but some of them are at least a bit off base. seo and search engine optimization also returned gay search engine.
• uses data from MSN Search, which gives them access to how people actually search, although they do not specify exactly how they use the data and how it is ordered.

Negatives:

• requires you to sign up for the Submit It / Bcentral URL submission service ($49)

• takes at least 20 minutes to find it and how to use it. I probably would not recommend using this tool much unless you like frustration.
• many of the steps in the Bcentral optimization process might be a waste of time and effort

ABC Tires:
Their tool only listed a couple useful direct terms, but also listed about 30 good related phrases or modifiers and about 50 useless ones. Their keyword research tool has a 90 result limit.

Search Engines See Also Search Results:
Many search engines including Snap, Clusty, Gigablast, Teoma, and Yahoo! Search offer alternate search suggesitions which may help you find related keyword phrases.

Good Keywords is a free downloadable keyword software tool which accesses the Overture search term suggestion tool and a few of the search engine see also searches.

Search Spy:
A variety of search engines allow you to see what people are currently searching for or buying. A few examples are Froogle, SearchHippo, Yahoo! Shopper, Kanoodle, MetaCrawler and a variety of others are listed on SearchEnginez.com.

Hot Searches, Google Zeigest, & Yahoo Buzz!

• Yahoo! Buzz Index - shows top movers and top searches in a variety of categories

• Google Zeitgeist - shows top news searches, image searches, and a variety of queries. Also shows top gaining and declining search queries.
• AOL Search: Hot Searches - browsable searches popular by generic category or amongst AOL members right now, in the past hour, or in the past day
• Ask Jeeves Interesting Queries - shows hot searches for the week
• Lycos Hot 50
• WordTracker also offers a hot search terms list
• Technorati offers top searches for the last hour
• TrendWatching is a popular site related to...watching trends

Word Relationships:

• WordNet lexical database

• WordNet search
• Lexical Freenet
• Thesaurus.com

Logfiles:
By tracking where your visitors came from you can extract more keyword combinations to focus on. Of course this only shows you where you already are, not what you are missing.

Internal Site Search:
If you have a large site you may want to use an internal site search to help people navigate your site. It also can help you find what you are not offering that you should be offering and what terms you should be targeting.

Run a Test Google AdWords Account:
It may be a bit more expensive, but the quickest and easiest way to get a bunch of keyword research data for a new site is to start an AdWords campaign for some broad matched generic terms in your industry and see what type of queries people are searching for. You can quickly add negative keywords if you are getting many untargeted visitors and harvest the referal data for more specific keyword phrases.

Why Deep Keyword Research is Important:
The specific terms have been changed, but the math comes from a live campaign. Here is some of the data from my ad groups related to ABC Tires:

Notice that as you add modifiers to target the search queries better the following occurs:

• the clickthrough rate increases (since the search and ad copy is more targeted to exactly what you offer)

• this higher clickthrough rate is seen by Google as increased ad relevancy, and helps lower the cost per click
• the conversion rate increases
• cost per conversion decreases

Deeper Terms are Usually Cheaper:
If you rely on only a few of the most generic terms for your traffic then competitors with deep pockets can easily wipe you out. Due to the clickthrough rate factoring into the click price only smart competitors who research their keywords well will be able to compete with a deep focused account full of hundreds or thousands of valuable search terms.

If a Term is too Generic:
The search term abc was hard to keep running, even with a ton of negative keywords Google did not deem the term relevant (due to low clickthrough rate) and disabled it after 6 clicks. They would prefer not to sell generic cheap clicks. If people search too generically they want them to refine the search so Google can make more money selling higher priced more targeted leads.

Google just stated that in a few weeks they are updating the AdWords system to accept less relevant search terms if people pay a higher minimum price for the ad. As a result I was able to bid on ABC, but in spite of little competition the premium for lack of relevancy meant that my bid price for it went up nearly as high as the bids for the more competitive terms.

Keyword Targeting: Exact, Phrase, Broad:

• Exact: [term one] only shows up when people search exactly for term one. Thus it has the highest relevancy, lowest reach, and sometimes a lower cost per click than the broad terms since it has a higher relevancy score.

• Phrase: "term two" shows up when people search for anything containing "term two" in the query
• Broad: term three shows up when people search for anything containing term and three and many terms that are semantically similar. due to it's broad reach it typically has the lowest clickthrough rate and may end up costing you more per click and drive up ad costs by charging you for some irrelevant traffic. It can also be used to help you mine keyword data though.

Keyword List Generators:
If you know common modifiers that people might use when searching for your products you can use a keyword list generator to help build thousands of keyword phrases in a matter of minutes.

I recently created a free open sourced keyword phrase generator. Before creating it my old favorite was The Permutator, which is downloadable software that costs $50, but I also think GoogEdit (free download) and this one (free web based tool) are also useful.

Misspelled Keyword Phrases:
This free keyword typo generator makes it easy to quickly generate misspelled related keyword phrases for your most common search terms.

Make sure you place misspelled words in their own ad groups / ad campaigns to make them easy to manage. Do not enable Google's dynamic keyword insertion ad copy with misspelled terms since Google does not want misspellings in their search result pages.

Even though I only recently added a bunch of misspelled terms to my clients account so far it has already converted and has a fairly low cost per conversion. It does not get a ton of traffic, but if almost all of the search result listings is irrelevant to the searchers needs and you are dead on target there stands a good chance of them clicking on your ad.

Keyword Research Tool Search Volume Accuracy:
I did not post about the accuracy of the predicted search volumes for the various tools for a variety of reasons.

• Even if there are many searches it does not tell you how related the searches are to what you sell.

• The tools will have volume errors, and the data is more for qualitative than quantitative research.
• If you are using the tools to build terms for pay per click campaigns usually the longer terms have lower prices and greater value. If you use ad group and campaign functions correctly it adds negligable time to expand out you keyword lists to thousands of more targeted terms instead of overspending on the top few generic terms.
• If you are unsure whether or not it is worth the effort to do SEO for a specific term you can always get a peak at traffic quantity and quality by running a test PPC campaign.

Hiring Keyword Help:
Dan Thies sells keyword research reports for $90, and also gives away a free keyword worksheet download and has a free video offering keyword research tips.

Pay Per Click Search Related Keyword Competitive Analysis:
Software such as AdWords Analyzer ($67) or Keyword Locator ($87) help you quickly get an estimate of how many competing AdWords advertisers there are for a given set of keywords, but do not offer deep analytical information the way some of the other tools listed below do.

HitWise starts at about $20,000 a month if you can afford it. It shows you what sites are sending traffic at competing sites and what search terms from each engine are driving traffic at competing sites from monitoring the internet traffic of 25 million web users. I was not too impressed with their Keyword Intelligence offering, but I imagine there is much greater value to the competitive analysis data. Some competitive analysis data can be had free or cheap though.

Spyfu is a free keyword research tool which shows you a small sample of terms that competing sites are ranking for in the search results and buying ads for. It cross references sites and keywords and top competing sites. It is not exceptionally in depth, but I might soon use it to add a few thousand contextually relevant terms to some campaigns.

AdGooRoo creates graphs of with frequency of Google AdWords display and ad position. They also monitor new competitors going into your keyword markets. Their service costs $2 per keyword per month, and they also give you 10 keywords that competitors are bidding on that are not yet in your account if you subscribe at the $99 per month level. They have a free PDF guide describing their services here. I believe Adgooroo is also adding features to track organic search results as well.

SEM Phonic is a new competitive research tool being beta tested right now which allows you to compare your URL to a few competing sites and industry related sites.

I just started playing with some of the competitive analysis tools, so pretty soon I will probably do a more in depth post on those.

2008 Update
We list many additional tool, including the SEO Book keyword tool on our SEO tools subdomain and SEO training section.

SEO Tools subdomain (free)

• keyword tools
• pay per click tools (see also Microsoft adCenter coupons)
• competitive research tools

SEO Training (member's only)

• keyword research training
• pay per click training
• competitive research training