Rank Modifying Spammers
My good friend Bill at SEOByTheSea has unearthed a Google patent that will likely raise eyebrows, whilst others will have their suspicions confirmed.
The patent is called Ranking Documents. When webmasters alter a page, or links to a page, the system may not respond immediately to those changes. Rather, the system may change rankings in unexpected ways.
A system determines a first rank associated with a document and determines a second rank associated with the document, where the second rank is different from the first rank. The system also changes, during a transition period that occurs during a transition from the first rank to the second rank, a transition rank associated with the document based on a rank transition function that varies the transition rank over time without any change in ranking factors associated with the document.
Further:
During the transition from the old rank to the target rank, the transition rank might cause:
- a time-based delay response,
- a negative response
- a random response, and/or
- an unexpected response
So, Google may shift the rankings of your site, in what appears to be a random manner, before Google settles on a target rank.
Let's say that you're building links to a site, and the site moves up in the rankings. You would assume that the link building has had a positive effect. Not so if the patent code is active, as your site may have already been flagged.
Google then toys with you for a while before sending your site plummeting to the target rank. This makes it harder to determine cause and effect.
Just because a patent exists doesn't mean Google is using it, of course. This may be just be another weapon in the war-of-FUD, but it sounds plausible and it’s something to keep in mind, especially if you're seeing this type of movement.
The Search Engine As Black Box
In ancient times (1990s), SEO thrived because search engines were stupid black boxes. If you added some keywords here, added a few links there, the black box would respond in a somewhat predictable, prescribed, fashion. Your rankings would rise if you guessed what the black box liked to "see", and you plummeted if you did too much of what the black box liked to see!
Ah, the good old days.
These days, the black box isn’t quite so stupid. It’s certainly a lot more cryptic. What hasn’t changed, however, is the battle line drawn between webmasters and search engines as they compete for search visitor attention.
If there are any webmasters still under the illusion that Google is the SEOs friend, that must be a very small club, indeed. Google used to maintain a - somewhat unconvincing - line that if you just followed their ambiguous guidelines (read: behaved yourself) then they would reward you. It was you and Google on the good side, and the evil spammers on the other.
Of late, Google appear to have gotten bored of maintaining any pretense, and the battle lines have been informally redrawn. If you’re a webmaster doing anything at all that might be considered an effort to improve rank, then you're a "spammer". Google would no doubt argue this has always been the case, even if you had to read between the lines to grasp it. And they’d be right.
Unconvinced?
Look at the language on the patent:
The systems and methods may also observe spammers’ reactions to rank changes caused by the rank transition function to identify documents that are actively being manipulated. This assists in the identification of rank-modifying spammers.
“Manipulated”? “Rank modifying spammers”? So, a spammer is someone who attempts to modify their rank?
I’ve yet to meet a webmaster who didn’t wish to modify their rank.
Google As A Competitor
Google’s business model relies on people clicking ads. In their initial IPO filing, Google identified rank manipulation as a business risk.
We are susceptible to index spammers who could harm the integrity of our web search results. There is an ongoing and increasing effort by “index spammers” to develop ways to manipulate our web search results
It’s a business risk partly because the result sets need to be relevant for people to return to Google. The largely unspoken point is Google wants webmasters to pay to run advertising, not get it for “free”, or hand their search advertising budget to an SEO shop.
Why would Google make life easy for competitors?
The counter argument has been that webmasters provide free content, which the search engines need in order to attract visitors in the first place. However, now relevant content is plentiful, that argument has been weakened. Essentially, if you don't want to be in Google, then block Google. They won't lose any sleep over it.
What has happened, however, is that the incentive to produce quality content, with search engines in mind, has been significantly reduced. If content can be scraped, ripped-off, demoted and merely used as a means to distract the search engine user enough to maybe click a few search engine ads, then where is the money going to come from to produce quality content? Google may be able to find relevant content, but "relevant" (on-topic) and "quality" (worth consuming) are seldom the same thing
One content model that works in such as environment is content that is cheap to produce. Cheap content can be quality content, but like all things in life, quality tends to come with a higher price tag. Another model that works is loss-leader content, but then the really good stuff is still hidden from view, and it's still hard to do this well, unless you've established considerable credibility - which is still expensive to do.
This is the same argument the newspaper publishers have been making. The advertising doesn’t pay enough to cover the cost of production and make a profit - so naturally the winner in this game cuts production cost until the numbers do add up. What tends to be sacrificed in this process - is quality.
NFSW Corp, a new startup by ex-TechCrunch and Guardian columnist writer Paul Carr has taken the next step. They have put everything behind a paywall. There is no free content. No loss-leaders. All you see is a login screen.
Is this the future for web publishing? If so, the most valuable content will not be in Google. And if more and more valuable content lies beyond Google's reach, then will fewer people bother going to Google in the first place?
The Happy Middle
Google argue that they focus on the user. They run experiments to determine search quality, quality as determined by users.
Here’s how it works. Our engineers come up with some insight or technique and implement a change to the search ranking algorithm . They hope this will improve search results, but at this point it’s just a hypothesis. So how do we know if it’s a good change? First we have a panel of real users spread around the world try out the change, comparing it side by side against our unchanged algorithm. This is a blind test — they don’t know which is which. They rate the results, and from that we get a rough sense of whether the change is better than the original. If it isn’t, we go back to the drawing board. But if it looks good, we might next take it into our usability lab — a physical room where we can invite people in to try it out in person and give us more detailed feedback. Or we might run it live for a small percentage of actual Google users, and see whether the change is improving things for them. If all those experiments have positive results, we eventually roll out the change for everyone"
Customer focus is, of course, admirable, but you’ve got to wonder about a metric that doesn’t involve the needs of publishers. If publishing on the web is not financially worthwhile, then, over time, the serps will surely degrade in terms of quality as a whole, and users will likely go elsewhere.
There is evidence this is already happening. Brett at Webmasterworld pointed out that there is a growing trend amongst consumers to skip Google altogether and just head for the Amazon, and other sites, directly. Amazon queries are up 73 percent in the last year.
There may well be a lot of very clever people at Google, but they do not appear to be clever enough to come up with a model that encourages webmasters to compete with each other in terms of information quality.
If Google doesn’t want the highest quality information increasingly locked up behind paywalls, then it needs to think of a way to nurture and incentivise the production of quality content, not just relevant content. Tell publishers exactly what content Google wants to see rank well and tell them how to achieve it. There should be enough money left on the table for publishers i.e. less competition from ads - so that everyone can win.
I’m not going to hold my breath for this publisher nirvana, however. I suspect Google's current model just needs content to be "good enough."
Comments
One of my biggest takeaways from studying ranking fluctuations over the past 6 months or so is that the amount of 24-hour change is shockingly high. Something like 80% of the Top 10 SERPs we measure change every night, to some degree. Some of this is algo updates (500+/year), and some is SEO/content (including QDF factors), but it's hard to believe that rankings are that dynamic unless they're actually being coded to change on purpose. I can't prove that, but it wouldn't surprise me at all to find out that some amount of noise is being injected into the system.
That way you stay inside Google and never go out, or so they hope. First Scrape, then become a content farm built on other publishers work, go Google.
"However, now relevant content is plentiful, that argument has been weakened"
Bit of a short sighted statement, no? The world isn't static. New things are happening all the time that have never existed until this moment - news storys, natural disatsters, world records. Google haven't seen ~25% of queries before which is astonishing, and means there's plenty of scope there.
...many of the sources of new content have over time become more beholden to Google.
Look at the market capitalization of Google (over $200 billion, with $40 billion in cash) & then look at the market capitalization of the New York Times (slightly over $1 billion, heavily loaded with debt).
Note that where publishers are not politically connected enough to have the local political body soak foreign tech companies, largely most of the mainstream media sites that have done paywalls have either failed at them, or have intentionally created paywalls that were porous.
About the only exceptions to the above are niche trade related content sources & pieces of the financial media, but even the financial media sites still tend to share at least teasers on articles and such. I think the Financial Times also gives like 5 free article views per month if you register a free account with them.
And if there is some source that is completely closed off from Google with vital market-moving information, someone will get behind the paywall & do a Huffington Post styled dumb downed rehash & own the story. Where there is demand people will quickly fill the supply chain & disintermediate the source if that source is behind a paywall.
I think this is yet more SEO's in conspiracy mode than anything. Google has to use factors such as links and citations to a site as a factor, or they would never truly be able to find relevant content to a query that users find useful. --> SEOs build the site authority --> Google gives them a decent rankinig --> Users bounce or find it less relevant than other content in the results --> Rankings drop. There very well could be a dual ranking system whereby Google is testing a new ranking level for the site, and if users don't like it, it drops.
I certainly agree that in many cases people want to externalize blame when something fails, so that might be the case quite often. And indeed poor user experience data can cause rank to drop (have seen hosting issues cause a listing to slide down the result set & then move back up as the host fixes the problem).
But in some cases some of these ranking impacts are largely outside the range in which meaningful user feedback is delivered. For example, I saw a listing where it was at about 90 or so. And then a bunch of link building was done. At some point the listing popped up to about #30 for a week or so & then it dropped right back into the 90s where it started ... and stayed there for months more.
Keep in mind that almost nobody looks on page 3 or 4 of the search results, so that rank change back and forth wasn't due to user engagement data.
Further, that same site a few moths later popped up to about #10, without changing the user experience at all.
That the rank changes are so drasting & are sorta step functions (rather than smooth & gradual shifts) would indicate that indeed Google is holding back on trust in some cases & then batch releasing it.
And this sort of behavior wouldn't be the first time it has been exhibited...
One interesting behavior with Google in terms of collecting end user feedback is that they place less weight on global link authority / PageRank in some international markets. In some cases a lower authority site that is somewhat aggressive with anchor text and/or on an exact match domain (EMD) might start ranking better in India & other countries before it ranks well in the United States. Another interesting side effect of that is if it picks up links in some other countries (due in part to having more exposure in those countries by ranking higher there) Google may sometimes localize those sites to those foreign markets & downrank them further in the global results (because they believe them to be local to those markets). Of course one can offset such localization inside webmaster tools, but the above certainly does point to the importance of user experience. However that example can also work poorly in terms of if the content is written for one culture & market and ranks first in another. I could probably write for hours about user experience stuff & potential implications of it, however even if it is counted significantly it doesn't mean that Google is unwilling to throw curveballs. After all, they did file for the above pattent, had issues with that ranking #6, have loads of different penalties & filters, and keep making search more complex with things like Panda and Penguin.
Hahah... I can see that evil look in that penguin
Dude .. I tink it was a slip of the tongue by google, its kinds when you get drunk and say things that you shouldnt have . But now atleast we know what googlew thinks about a lott of people and its not looking good..seriously
Now they want to release one more update ...are you kidding me !! so many years of hard work... all gone !!
I was stitched up by google and so are many others.. let People do what they want when the goings good ..w hen it becomes a bit too much and you have made your money.. just swing a hammer randomly and forget about the innocent that get hit
cheers
jeff
It's funny when you think about it, but if Google has 37% of search instead of pretty much all of it,, all this noise and kerfuffle wouldn't be heard. People wouldn't mind about the changes because overall impact on their business could be offset by working on the other potential search revenue generating channels. Wouldn't it be nice for SEO if Bing just worked the same way it did when paid inktomi and a looksmart directory did. Or if Google was more DMOZ heavy like the good old days.
But the unfortunate fact is that they do own search, and that means that every little change has a huge potential backlash of unhappy webmasters. People in India up in arms about the changes not to mention all the "we've been devastated and never even did any link-building!"
If Google had a search that really was as cracked up to be in terms of its public blog broadcasts, then we could all site back and relax, safe in the knowledge that your skills in marketing would result in your superb websites coming out top, simply because they were the best. But that simply doesn't happen. I can't remember if it was four or five years after Google said they were able to detect JavaScript and hidden text that it was still being used by some of the biggest names in the on line space, with superb results.
The Achilles heel of Google got well and truly stretched for many years and it was to the benefit of a whole industry that got setup literally of the back of this weakness, links.
Another mistake they made was simply to draw the black hat white hat argument into the open. Sorry but business didn't really care. Whereas before you'd go on a pitch just talking about results, now you would simply do the same pitch and say, If you want to be there in a week it will involve xy and z with the following risks. Judging by what industry professionals (hi rand) say about Xrumer I guess they've just used it for forum spamming when in fact there is no other link analysis tool on the planet that comes close to it. But please keep telling everyone it's no good.
But hold on and take a step back and look at the chronology of what went from being a legitimate way of getting traffic to pages through to those same techniques being considered black hat. If the way that Google got coverages was also part of the increase of services becoming available to make ranking possible, then this in the some way was beneficial to both parties. The SEO provider because of an income, and the search engine by nature of the fact that there were services that made good the statement that Google was the only search engine that was worthy of any consideration. A bit of quid pro quo in my book.
But no, what you have is a situation of waking up one day and finding that what was being told to you was the best way to rank and that now all your clients are sitting on page 4. Let me put it another way. One day you are told that your sales pitch must involve convincing people that petrol makes the cars go, and the next that in fact it wasn't petrol and that there really but diesel and there was little that could be done about the damage, sorry about that. That's why I advise any SEO business to seriously consider adding a term to their Ts&Cs which basically says that the delivery of the service is in line with current search engines guidelines at the point of delivery and that if those search engine guidelines should change after delivery that you charge your client a daily rate to remedy any changes.
So now we are that juncture where what Google wants to do is make everything paid. Turn the concept of the Internet as that cool new place to go and get something that you wouldn't ordinarily find, into QVC. This has already happened in the financial sector (and can be see by simply running a few queries and pulling back the unique number of domains in the first 100 results. Just take all the top performing keywords and run that through Google, you will be surprised at how few different websites come back) and I imagine others will see it in theirs.
What'll happen is Google will steam ahead until such time as someone that can says they can't do what they are doing.
Or they'll just loose market share and no-one will care.
:)
...thanks for the great contribution :)
Well all my suspicions, fears, even my gripes are here in your expert comments lain out like neat rows of lush green turnips. Despite the wonder and need of having chlorophyll fill those fibrous leaves of life giving vegetation, the Sun still failed to shine this morning and all the mornings since Google went coo coo.
Regardless of the pain of the little publisher, and our little gnashing teeth even after having been good SEO children, I still appreciate all your diligence and study on the matter. My question is, "When do we file the lawsuit?" ;)
Always,
Phil
"Or they'll just loose market share and no-one will care."
One can only dream. Eventually, that will happen. But the question that matters is "when?" I fear we still have a long road ahead of us.
That said, however, G is getting pretty cocky lately with various gov't agencies. Maybe they'll expedite the process internally.
Then again... maybe we should be careful what we wish for. I have a feeling that the future alternatives to Google are likely going to lean towards the "QVC" approach rather than the granola-organic-everyone-is-special approach to ranking content.
Once again, the smbiz community is at the bottom of the hill. And we all know what rolls downhill...
-Chris
As a white-hat SEO in a competitive market, I am glad about the “transition rank” patent because it will make my job that much easier. Oftentimes, my link building research is clouded by spammers. The usual prospecting tools don’t work very well because their indexes get filled with spam, so I turn to Google, which can even be heavily saturated with affiliate spam for some of my keywords. As an example, this query, “site:.edu insurance company” returns spam for several pages of SERPs. If transition rank is already in use, then my hope is that Google expands on this choice.
Also, I doubt that the purpose of transition rank is to mess with SEOs, and my hunch is that there is a much more data driven purpose. For example, Google could learn that the page transitioned to spot 45 loses 100% of its traffic, which would mean the site has very few return visitors, and that could be a signal to Google of low quality content. This is only one example, and I am sure there are other possibilities. The rank transition is a controlled experiment that will tell Google more about a website. I think Google cares a lot less about SEOs than we like to think. =)
versus
one of these things is not like the other.
publicly Google claims that there is a big distinction between SEOs & spammers, but the 2 are synonyms in the above patent.
If your link building research is being clouded by spammers here's a link that I think you can find helpful, because with it you can pretty much filter out whatever you want: http://www.googleguide.com/advanced_operators_reference.html
On a more general note however my advice is to be careful because history shows that what Google is really interested in is maximizing revenue and cutting out the agency middleman (and that means you) so they can handle advertising relationships themselves.
What do you think happened to all those e-comerce clients going via agencies who implemented "black hat" linking strategies (the same strategies that was supposed to be the ranking feature of the Google algo, and which have had the same change in reclassification appplied as that of Doorway/Hallway pages in previous versions of Google's slapping of SEO's). They all went crying to Adwords. And the advertising agencies just lost some of the great margins of SEO, perhaps a few clients, but essentially nothing really changed. As for the little guys, well do a search "why my site has disappeared from Google" over the past year to get a lookin :).
But that's fine and okay with me because what Google is doing is frankly what any business would want to do and I'm not getting hung-up on these changes or getting down in the doldrums because of it. Personally I'd like to see them get a big Government slap-down just because of where they took their business, but again not loosing any sleep over it.
However what I do have is sympathy for the large swathes of the SEO community that still cling to this idea that if "I do what Google" says, "I will be rewarded". It doesn't work like that. You have to be cleverer. I've had the luxury of being able to view the SEO landscape through it's evolution these past 12 years and the various hypocrisies within it, as have many others with me.
But then some people like to come into work each day and do the same thing, and that's not me, so this industry is more than perfect for me.
thanks for sharing :)
Look what you've created:
http://www.zdnet.com/any-seo-could-damage-your-site-as-a-spam-site-70000...
:)
Google throwing off results
I once tried making my own web-based rank checking tool back in 2007, it was more of a scraper, not using any APIs and was made in PHP. I didn't even use cURL (http://php.net/manual/en/book.curl.php) I simply used the PHP function file_get_contents and it was running well while testing and programming and was done by the next day.
After checking the script the next day, it was still working but the results seemed incorrect. Then when I waited more than a week to a month, I noticed my results were not changing at all. And looked way different from the SERPs.
Now one thing to note is...
1) My script had no caching system in place.
2) The results were the same even in different views on different ISPs on different days so it is not something cached on my ISP.
3) The function really pulls the source code of a Google SERP page, and if a connection is not successfully establish, the result will be blank.
4) The results seemed to purposely been custom to my script alone that the results will never change. But instead of blocking my script, Google purposely kept pushing a page to my script but with false rankings.
Ever since then I said, I am not going to make my own rank checking tools, so much to do... from modifying user agent, rotating IP addresses, adding in pauses to emulate human behaviors, etc.
I looked at the patent on the link above with the 2012 date, but it was filed in 2005. And I think I have seen this in action already when I was trying to make my own tool in 2007.
They have also done the same thing to some PageRank scraping tools. Rather than returning null & letting the person know that they needed to fix it, sometimes they would skew the number a bit high & other times a bit low.
I recently wrote a rank tracker in PHP with cURL and it worked fine (until Google changed things a few months later). After dissecting the SERPs I found that it possible to fix the tracker to get it to work again. So I recommend trying again and using cURL instead.
This line made me chuckle...
"The systems and methods may also observe spammers’ reactions to rank changes caused by the rank transition function to identify documents that are actively being manipulated. This assists in the identification of rank-modifying spammers."
Red herring… see what people panic and change – THAT'S what they have control over. THAT'S what they're doing to attempt to influence their ranking. Haha. God I love Google.
Great Article. comments - I learned a lot from the insight . Does anyone know where my assistant could get access to a blank Acord 37 copy to edit ?
Add new comment