We recently added an SEO X-ray feature to SEO for Firefox. You must use Firefox 3.0 or above to see these features, but if you want to see...
how the on page optimization of any page looks (headings, meta description, page title)
the keyword density of the page and popular phrases on the page
how many links point into a page (total links, or links from external resources)
how many links point out of a page (as well as the anchor text of these links, nofollow vs follow, internal vs external - all exportable in CSV format)
then this new feature makes it quick and easy to do all of that. Simply right click on the page you are viewing, scroll down to SEO for Firefox, and click on SEO X-ray.
That will show you an overlay on the screen like this
We are planning on doing another update in the next couple days, and may add...
the IP address of the site (and links to other sites on the same IP address)
character and word counts for page title and meta description body content
a link to the domain tools overview page for the associated site
If you are using Firefox 3 and SEO for Firefox please give this a try and let us know what you think.
So my timing on that last post was a bit off, but I still think the general thesis is valid. But now that there has been so much negative feedback I figure it is my job to play devil's advocate and highlight reasons why most SEOs do not need to be too worried about Linkscape.
Cool Features
Unique Linking Domains
One of the coolest features of this tool is knowing the number of unique linking domains pointing links at a specific site, but that feature is for paying members only.
A competing tool by the name of Majestic SEO allows you to see that data as part of their free overview. Click on the image below for an example.
If your competitor has high authority links then you need more than just quantity to compete, but if most of their backlinks are garbage then this is a good stat to have, along with many other stats you can get from tools like SEO for Firefox.
Spam Reporting
Not that I advocate spam reporting (as the official guidelines have departed from reality so much that almost everyone that ranks is spamming and/or spammed in the past to get to their current market position), but for professional SEOs that own dozens of sites and like doing spam reports to Google this might be a good tool for outing competitors, since it makes it easy to find some noscript links, links from off the page, inbound 301 redirects, but the average webmaster probably does not need to worry about that.
A Bit Top Heavy
One of the biggest limitations in Linkscape is that you can only go 500 results deep unless you want to buy a custom report. They allow you to see various lenses of 500 at a time through search features and filters, but a big recommendation I can make on this front is for them to allow you to see all that data, even if it requires exporting data to CSV...they already spent the money to collect the data, so if you're a customer they may as well give it to you...it helps nobody if nobody sees it.
Majestic SEO appears to have a similar sized database as Linkscape, and they allow you to do a full data export for your own domain free of charge. Other domains they charge a scaling price for depending on the number of links to the domain.
More Cool Features?
Nick Gerner promised more features in the next version of Linkscape, but unless they start buying usage data and become more like Compete.com I am not sure if it will be a game changer. On to explaining why...
Personally, I'm not too worried. You want to compete with me and get links in places where I'm listed? We get listed in places where editorial rules. So just knowing where we're at doesn't get you in the door -- you have to be good enough to walk in. And if you are good enough, well, good I guess.
The highest quality links typically tend to be editorial in nature, with many of those being driven by social relationships. No matter how much one decides to analyze link patterns, they can't re-create most of the link relationships if they don't already have the content quality, market exposure, and awareness. And if you copy someone's idea after they already did it you need to greatly improve upon it to get credit for it.
2. Tons of Alternative Data Sources
Common link analysis questions...
How do I Get a Basic Competitive Overview of the Search Results?
Search Google with SEO for Firefox turned on. Make sure you are pulling data in the automatic mode while searching.
I Want to do Anchor Text Analysis. How do I Analyze Links?
Some options include...
SEO Link Analysis - a free Firefox extension that adds anchor text to Google Webmaster Central and Yahoo! Site Explorer.
Backlink Analyzer - a free desktop based tool I had created a few years ago that pulls data from the Yahoo! API. Make sure to watch the video on the download page before using it.
I Want to Find New Links to Competing Sites
If you want to find what someone's best ideas are all you have to do is subscribe to the Google Blogsearch feed for links to their site, like so. That should list many of the people who are talking about this site.
A paid option on this front is Advanced Link Manager. It costs $199 (or $299 if you package it with Advanced Web Ranking) and scrapes data from Yahoo!, keeping track of the date when the link was found.
I Want to Find New Links to My Site
This is the same as competing sites, but you can also use your web analytics and server logs to dig up additional information. You can also look inside Google Webmaster Central to download backlink reports.
I Want to Find The Most Authoritative Links Pointing at a Site
Yahoo! Site Explorer generally orders backlinks roughly in terms of authority, with some of the most authoritative backlinks showing up at the top of their results.
I Want to Get an Estimate of Unique Linking Domains
Majestic SEO offers a free estimate...though, like LinkScape, their crawl is not as comprehensive as Yahoo!'s.
I Want to Find Hub Links?
Hub Finder is a great tool for finding topical hubs.
Google TouchGraph is another great option with a cool graphical interface.
What Sites Drive the Most Traffic to My Competitors?
The best way I have found to get this data is from Compete.com Referral Analytics, though it requires a $500 a month subscription...which is a nice chunk of change, unless you are already doing quite well!
Do I Have Any Broken Links?
Xenu Link Sleuth will crawl your site and help you find broken links, showing you which pages the broken links were on.
Each search engine has its own crawling priorities and own web graph. Google has probably spent hundreds of millions of dollars building and refining their crawling sequence. No two crawls are the same.
4. Yahoo! Search Counts Link Weight Differently Based on Page Segmentation
Google's PageRank was designed based on a random walk theory, where browsers click a random link on the page. But search engines are looking to move beyond the random walk model.
The irrelevant links at the bottom of a page, which will not be as valuable for a user, don’t add to the quality of the user experience, so we don’t account for those in our ranking. All of those links might still be useful for crawl discovery, but they won’t support the ranking.
5. Microsoft May be Looking to Heavily Incorporate Usage Data
Microsoft did research on BrowseRank, which aims to use actual usage data to augment (or perhaps replace) their link graph. Be default, Internet Explorer 8 sends usage data to Microsoft...when you know what 80% of web users are doing you do not need to rely on a random walk.
Think of having access to the majority of the web's usage data like this:
If Google's algorithms are more relevant than Microsoft, then putting weight on usage data allows Microsoft to quickly catch up by weighting whatever Google is weighting
Microsoft could theoretically be better than Google at filtering out paid links, as most paid links in a sidebar or footer do not send much traffic...and thus could easily be weighted less than links in content - though with Google owning so many products they could improve significantly on this front as well, if they decided to use their AdSense data, analytics data, Chrome browser data, Feedburner data, and toolbar data.
Beyond those editors there are many search engineers inside the webspam team offering a variety of techniques to throw off SEOs, including
stripping all PageRank from a site and killing all its rankings
stripping some portion of a site's PageRank and ranking abilities
stripping PageRank from the toolbar but still allowing sites to rank
showing full PageRank in the toolbar, but killing the ability of a link to pass PageRank
Without working inside of Google and/or buying and testing lots of links across a wide array of sites and verticals it would be hard to know if any particular site passes PageRank, and how much it might pass. For instance, a link from Text-Link-Ads.com's website is one of my highest MozRank links, but I doubt Google places much weight on that link since Google does not let Text Link Ads rank for their own brand.
7. Search Engine Editorial Policies are Selective, & Constantly Changing
According to Udi Manber, Google did 450 search algorithm updates last year. Even if you could somehow catch up with all the editorial stuff search engines were doing to manipulate their version of the link based web graph, you would have a hard time of keeping up with it - let alone accounting for the hoards of usage data the search engines have.
The status of a link (and its ability to pass PageRank) may arbitrarily change based on media exposure. In the past many websites were hijacked by 302 affiliate links (this even happened to Google's site, and this is still happening today to corporate sites as big as Snapnames).
Shockingly, when asked point blank if affiliate programs that employed juice-passing links (those not using nofollow) were against guidelines or if they would be discounted, the engineers all agreed with the position taken by Sean Suchter of Yahoo!. He said, in no uncertain terms, that if affiliate links came from valuable, relevant, trust-worthy sources - bloggers endorsing a product, affiliates of high quality, etc. - they would be counted in link algorithms. Aaron from Google and Nathan from Microsoft both agreed that good affiliate links would be counted by their engines and that it was not necessary to mark these with a nofollow or other method of blocking link value.
A few years ago I set up my affiliate program to use 301 redirects to prevent hijacking, and get any link benefits I could. But right after I changed by business model to a membership site my affiliate program was featured/outed in this interview, and it no longer passes PageRank.
Watch the above video and see how at 2 minutes and 15 seconds in my site was put up for review to any Google engineer that happened to watch it.
The same set of links, to the same site, using the same format, under similar circumstances...
counts for most major corporations (and is allegedly an approved and legitimate strategy)
counted for this site for years
stopped counting around the time they were outed by a popular SEO blogger
8. Temporal Algorithms + Domains Expire, & May Lose PageRank
Search engines may place weight not only on the number of links pointing at a page, but also on the rate at which links are accumulated. Even if you know the raw number of links and the site age it still does not tell you how many links were built last month or in the last year.
Not only are links born, but some of them rot. The web graph as a whole is over a decade old. Linkrot was a big issue in 1998, and it is still a big issue today. In 1998 6% of links were broken, and the DotBot crawl shows 7% of links being broken.
To appreciate how bad linkrot is...
if you publish a large site with many outbound links, run Xenu Link Sleuth through your site and see how many broken links you find
Some domains that expire may keep their PageRank, but many expiring domains lose their PageRank. With how hard it is to build links today and 1 in 7 links broke there are SEO tools designed around trying to capture this link equity
The domains that die off may later be re-registered and re-purposed. And keep in mind that the 1 in 7 broken links number is actually much higher than that when you consider how many people buy expired domain names and build them out.
By creating an index of the web in 2008 a person would have no idea if...
the links occurred recently
if the links are old
if the site expired and potentially lost much of its link weight
And Matt Cutts generally hates re-purposing expired domain names. Why? The very first spam site he found was a high PageRank expired domain linked from the W3C. That site was converted to a porn site, and ever since then (before Matt was the head of the webspam group - before Google even had a webspam group) Matt has not liked expired domains.
Matt offers background on that story 30 seconds into this video:
9. Advancing Algorithms That Move Away From PageRank & Anchor Text
domain history (ie: spam infractions/penalties, etc.)
site authority
signals of locality (hosting location, TLD, link sources, etc.)
searcher intent (Google's Amit Singhal stated "the same query can mean entirely different things in different countries. For example, [Côte d'Or] is a geographic region in France - but it is a large chocolate manufacturer in neighboring French-speaking Belgium")
other forms of search personalization (past searches, user subscriptions, frequently visited sites, etc.)
editorial partnerships with news companies & other universal search categories (like Google Shopping Search and the maps local onebox)
usage data (especially with sites they host, like YouTube)
All of those links might still be useful for crawl discovery, but they won’t support the ranking. That’s what we are constantly looking at in algorithms. I can tell you one thing, that over the last few years as we have been building out our search engine and incorporating lots of data, the absolute percentage contribution of links and anchor text to the natural ranking of algorithms or to the importance in our ranking algorithms has gone down somewhat.
Final Thoughts
It is not that Linkscape is a bad tool, it is just aiming to do something incredibly complex, and as long as Yahoo! Site Explorer gives us a decent free sample (and other tools let us layer data on top of Yahoo!) we can get a good idea of the approximate level of competition for free. But with Yahoo! at $12 a share, if Yahoo! gets bought out and Site Explorer goes away then Linkscape (or Majestic SEO, depending on who does a better job of innovation) might be one of the best SEO investments one can make.
Compete.com quietly launched a referral analytics product as part of their advanced package ($499/month). Even as a free user you can see the top 3 results for any site, which can be used to see how reliant a site is on search. Why is % of search traffic an important statistic?
If search traffic (as a % of total traffic) is low (relative to other competing sites) then it could indicate that there are organic optimization opportunities that are currently being missed and/or that site has a large organic traffic stream that can be marketed to in order to help it improve any search related weakness.
If search traffic (as a % of total traffic) is high (relative to other competing sites) then it could indicate that the site is near its full search potential, that the site is not very engaging, and/or does not have many loyal users
Here are search stats for SEO Book. Note that Google controls a minority of the traffic to this site, which means they have limited direct influence on the revenue of this site. Some sites are closer to 90% Google, which makes it easy for Google to effectively remove them from the web!
This sort of data is important for considering the viability of a business model, the stability of a site, and what multiple a site should sell for. It can also be used when considering the CPM of an ad unit - search traffic is much more targeted and goal oriented than a person browsing a forum is.
Until everyone and their dog started looking at PageRank (and how to manipulate it) it was a rather sound way of finding the most valuable backlinks. But with the pollution of endless bought links, nepotistic links, and PageRank only being updated quarterly it is tough to glean much market data from only looking at PageRank. Tools like SEO for Firefox (especially when used on a Yahoo! backlink search) allow you to gather more data about the quality of link sources. But they all try to measure proxies for value rather than how people actually surf the web.
Microsoft BrowseRank research would use browsing data to supplement PageRank on determining relevancy. In Internet Explorer 8 (currently in beta) a person's browsing details are sent to Microsoft by default. With ~ 80% of the browser market, Microsoft does not need to use a random walk for the core of their relevancy algorithm - they know what people are actually doing, and can use usage data as a big part of their relevancy algorithms.
Using a tool like Compete.com Referral Analytics makes it far easier to poach top affiliates, discover the best ad buying locations, and replicate a competitor's best backlinks. Be forewarned that the tool only works at the domain level, so it is much better at analyzing Yahoo.com than shopping.yahoo.com.
Along with referral analytics Compete offers destination analytics, which let you know what websites people visit AFTER visiting a particular site...which should help you glean information about how sites are monetizing, what offers are working well, what sites are well referenced by another site, and what sites people go to if they can't get what they want on the current site.
At $500 a month, this tool is probably only going to be used by those who are already fairly successful rather than as an entry level tool.
Google has been changing the code used to display their search results a number of times over the past couple days. We recently updated SEO for Firefox and Rank Checker. Both should work as of now, and if any more SERP changes happen we will try to update the extensions as soon as possible.
If you need to update these tools you can do so within the Firefox browser by clicking on Tools in the top Firefox menu, then from the Tools menu click on Add-ons. This will pop up the Add-ons / extensions window. At the bottom of this window there is a Find Updates button you can click. That will bring in the new updates and then when you re-start the browser the extensions should be fully functional again.
James from Semvironment created a plug in to automatically email webmasters you link to from within Wordpress blog posts. When he launched it, the opening post sent me an email
Hi! We linked to your website in our post: Link Builder for Wordpress - Download it Now!. Please stop by and check it out, subscribe to our blog and if you find something useful on our site or blog - we would welcome a link back anytime ... no obligation - we're happy to link to high quality websites and blogs like yours! To your continued success, [Your Name Here]
That link was broken (pointing to a revisioned archive version of the post before the URL changed), but even beyond that I sorta do not like the idea. Why? Automated communications is the enemy of relationship building. And the worst people to offend are the people you find interesting / important / influential enough to want to talk about them. You can get the person's attention just as easily by clicking the link in the post a dozen times and by commenting on their blog. And most of them would even be up for lending their time to you if you stoke their egos.
As more people start using a wider array of automated link building tools the effectiveness of automation will drop. And you can't train an automated tool to be personal. A tool like this might work in some verticals for a year or two, but if people find it effective look out for the tragedy of the commons to be heading toward your inbox some time soon!
Google announcedAdPlanner, a tool to help ad agencies find where desired demographic audiences are active online. The WSJ highlighted how the new Google tool can help make the ad marketplace more efficient:
The Web-audience data could be combined with the ad-serving system, so that advertisers would be able to find out whether they would reach the right audience before they committed to placing an ad.
In addition to AdPlanner, Google will launch another tool that compares consumer response to ads against a control group of users who did not see the ad:
Separately, Google this week is expected to roll out a new tool aimed at showing how Web surfers respond to online ads. It will compare groups of people who are exposed to an ad with others who haven't seen it, taking into account such factors as search activity and site visitation.
Our CMS only shows 300 comments per page. I am sure there is a way to enhance that, but for ease of publishing and following the conversations I decided to close the last SEO for Firefox thread and start a new one. If you have any questions about SEO for Firefox this is the place to ask them. :)
Summize is a conversational search engine which allows you to search Twitter in realtime. Useful for finding customer feedback even when people do not provide it directly to you. For example, I just found out that for some people the Rank Checker Firefox extension stopped working after the last update. So I just reverted the extension and am awaiting another update from the developer. Summize offers RSS feeds so you can track conversations mentioning your brands and/or important topics.
Summize offers an API which can be used to generate free content for your sidebar if you publish Mahalo-like content, though that is a bit spammy. ;)
