Why Was Demand Media Torched by Google? Branding

We Are Not a Content Farm

When Google began speaking publicly about content farms Demand Media's Richard Rosenblatt stated that it would be silly to call their stuff a content farm & he emphasized the quality of their content & care that went into it. Of course, those who bothered looking at the content often saw something different

Panda II Hits Demand Media

When Google did the global roll out of Panda earlier this month, they also modified their approach to core Panda algorithm to include user block data:

Today we’ve rolled out this improvement globally to all English-language Google users, and we’ve also incorporated new user feedback signals to help people find better search results. In some high-confidence situations, we are beginning to incorporate data about the sites that users block into our algorithms. In addition, this change also goes deeper into the “long tail” of low-quality websites to return higher-quality results where the algorithm might not have been able to make an assessment before. The impact of these new signals is smaller in scope than the original change: about 2% of U.S. queries are affected by a reasonable amount, compared with almost 12% of U.S. queries for the original change.- Amit Singhal

While many of Demand Media's sites got dinged in the first update, the fall of content farms in general meant that any site operating in that space which was not hit ended up seeing a sharp increase in traffic (as so much of the competition fell). As sites like AnswerBag and Livestrong fell, eHow's traffic increased significantly. I believe Google didn't want to rely on end user block data because it would make it easy for people to do competitive sabotage, however I think they needed to use it in order to hit eHow with the update. eHow had a number of signals (some older quality content, nice web design, syndication partnerships, tons of media exposure, etc.) which made it hard to whack it without creating too much collateral damage unless the block data was used.

Demand Media's Google Traffic Off 40%

Forbes.com highlighted Hitwise data which estimated that Demand Media traffic from Google is off 40%:

In the first two weeks of January, 0.57 percent of those who departed Google next visited a site operated by Demand Media ... by mid-April, with the full suite of Panda updates in place, Demand was feeling the pain. As of April 16, it accounted for only 0.34 percent of Google’s downstream, a 40 percent decline from the start of 2011.

Demand Media's Stock Falls 40%

Incidentally, over the past couple weeks Demand Media's stock is off roughly 40%

With that in mind, let's consider why eHow got torched. Here is a visual interpretation of the rise & fall of content farms. Here is part 1 of the eHow story, and part 2 follows below.

Branding

Ultimately I believe if content farms did not market themselves as sleazy operations almost nobody would have noticed or cared. You didn't see many people talking about "the content farm problem" until after Demand Media was featured in Wired as the cheap, disposable answer factory.

That article not only inflamed journalists (who were losing their jobs due to downsizing, outsourcing, and technology changes), but also inflamed anyone who created original content and later saw a rewrite of their own work replaced by eHow.

That article (which claimed eHow to be profitable as hell, a fuzzy claim depending on how one accounts for content depreciation) was aimed at trying to position Demand for an IPO and to try to pull in more media syndication partnerships.

What it did was inflame the web community & encourage others to play the same game & create content farms based on the blueprint Demand gave away. When a piece of marketing either pisses off almost everyone & encourages many of the people who are not pissed off to compete directly against you & cut your margins it is not a successful marketing approach.

Wages

A reason it was so easy for journalists to claim bad things about Demand Media was that the wages were so low that they didn't practically allow for any in-depth research to be done (unless a person was willing to work far below minimum wage). Thus when journalists started to dig into eHow's business model they got eHow writers to state things like:

"I was completely aware that I was writing crap," she said. "I was like, 'I hope to God people don't read my advice on how to make gin at home because they'll probably poison themselves.'

"Never trust anything you read on eHow.com," she said, referring to one of Demand Media's high-traffic websites, on which most of her clips appeared

Scale

The larger your scale is the easier it is to find something wrong with what you are doing. 1% of a really big number is much greater than 10% of a rather small number. If you are cutting corners & operating at scale & create a lot of enemies then I wish you the best of luck, because you are going to need it!

Outrageous Content

In spite of letting a few things fall through the cracks, to this day there are some OUTRAGEOUS eHow titles. A friend showed me a couple and after 5 minutes of searching I found:

Nose Picking

  • How to Pick Your Nose The Proper Way ehow.com/how_5722363_pick-nose-proper-way.html
  • How to Pick Your Nose or Scratch Surreptitiously ehow.com/how_2181862_pick-nose-scratch-surreptitiously.html
  • How to Effectively Pick Your Nose ehow.com/how_5067366_effectively-pick-nose.html

Exploring Other Orifaces

  • How to Fart ehow.com/how_2151823_fart.html
  • How to Stop Farting ehow.com/how_4785860_stop-farting.html
  • How to Muffle a Fart ehow.com/how_2320127_muffle-fart.html
  • How to Poop in the Woods ehow.com/how_2179463_poop-woods.html

Productivity Advice

  • How to Not Get an Ehow Article Erased ehow.com/how_5570908_not-ehow-article-erased.html
  • How to Slack at Work (and not get caught) ehow.com/how_4522164_slack-work-not-caught.html
  • How to Slack Off at Work and Not Get Caught ehow.com/how_4837878_slack-off-work-not-caught.html
  • How to Do Nothing at Work and Still Get Paid ehow.com/how_4430256_do-nothing-work-still-paid.html

Honing Your Social Graces & Charm School

  • How to Manipulate People to do Your Bidding ehow.com/how_2167832_manipulate-people-do-bidding.html
  • How to Get a DUI ehow.com/how_4825159_get-a-dui.html "You might think getting a DUI is as easy as getting behind the wheel of a car after drinking alcohol. But that's only half the battle. You also need to get pulled over by law enforcement and cited for it."
  • How to Not Be a Husband Caught Cheating ehow.com/how_5528899_not-husband-caught-cheating.html
    "Don't leave trails which can and will turn into signs you're cheating. First point to remember is to not use any computer your partner has access to when you communicate via email or IM to the cohort."

AdSense Click Fraud

  • How to get banned from Google Adsense ehow.com/how_5740892_banned-google-adsense.html
  • How to Increase Your Click Through Rate with Google AdSense ehow.com/how_5203081_increase-through-rate-google-adsense.html
  • How to not get Caught With Google Adsense Click Fraud ehow.com/how_5979999_not-caught-google-adsense-fraud.html

Leveraging Expired Domains

Demand Media bought out a leading domain registrar named eNom & leveraged some of the expired domains with links to prop up eHow, by 301 redirecting those domains into eHow's deep pages.

Javascript Nofollow on Outbound Links

Most of eHow's outbound links were coded in a javascript that prevented search spiders from being able to credit the original content sources which Demand Media writers used as the base for writing their content.

If you throw off links you get some love for it from your fellow webmaster, but no publishers like a PageRank black hole (unless they own it).

Duplication & Auto-generated Content

eHow was not only churning out loads of shallow content, but Demand Media was also using the data gleaned from eHow to make sister sites which included auto-generated pages and feeding search engines their own results.

They Made Google Look Stupid

Doing one thing and claiming another can provide cover for some finite period of time, but ultimately when you create such a spectacle out of Google that your exploitative ways become the core marketing message for Google's competitors you know your days are numbered. And given that the Wired piece made the media hate Demand Media, there was nobody left to defend them other than folks who would also seem in some way conflicted.

Ultimately this goes back to the core issue that hurt Demand Media: branding.

Don't make Google look stupid. That is the #1 rule of SEO.

Why Content Farms Are Here to Stay

Much noise was made recently about Google taking a whack at so-called content farms -- sites which apply industrial production techniques to the creation of content targeting the long-tail of the query distribution. This is a subject of huge interest to many Internet businesses, either because they advertise on the AdWords Content Network (and, by extension, on content farms), because they compete with content farms on particular searches, or merely because they hate seeing content farms in their search results. As luck has it, I am three for three. It pains me to say it, but content farming is here to stay. It is an economic inevitability.

The Attention Economy

Much of the Internet currently operates in an attention economy, a level or two removed from direct monetization. Facebook is worth in excess of 50 billion not just because they're making money hand over fist -- though they are -- but because they have achieved a dominant position in the attention economy, and they command such huge rivers of attention that they can trade trickles of it to people for actual money.

Google is the dominant player in the attention economy -- they harvest vast amounts of attention via controlling navigation on the Internet (via a commanding lead in search), they sell attention in the form of AdWords ads, and they provide a marketplace for attention with their AdSense product.

Individual publishers -- from the New York Times down to the smallest hobbyist site on the Internet -- are also largely in the attention economy. For a mega-brand like the New York Times, attention can be generated -- they can literally make news. Disney has a repeatable industrial process which takes as input one female teenager and produces as output a cultural phenomenon with hundreds of thousands of rabid fans.

Smaller players -- Google back in the dorm room days or hobbyist sites today -- largely cannot create attention on these scales, they can only harvest attention which already exists. Attention exists in the world for things independent of their own existence. People play golf. People bake cookies. People read Dan Brown novels. People receive massages. For all these things and more, people demand content: they want to improve their golf swing, they want new cookie recipes, they want new Dan Brown novels, they want massage how-to videos. And they are willing to pay with attention, a scarce commodity which can be converted into cash.

The Economics of Content Creation

Consider a hypothetical Internet with no efficient way of converting attention into money. This is not difficult to imagine: it was essentially the Internet of the dot-com bubble, where everyone wanted "eyeballs" but "eyeballs" plus banner advertising resulted in economically non-viable businesses. In this hypothetical Internet, content is mostly produced by people who have intrinsic reasons for creating it: hobbyists who want to share their passion, law professors who want to increase their professional reputation, governments who need to employ somebody and might as well employ a webmaster, and the like. This is widely viewed as a Garden of Eden scenario: the Internet, without the corrupting influence of money.

We had this Internet, and the average user experience was miserable.

Ability to publish content on the Internet was once dominated by presence of arcane technical skills (being a "webmaster", a title which thankfully has fallen out of fashion). Webmasters were, by and large, very geeky people. They largely scratched their own itches, which (predictably) resulted in an Internet chock-full of Dungeons and Dragons character sheets, trivia about Matter-Eater Lad, and fansubbed anime episodes.

Less well-represented on the Garden of Eden Internet was content appealing to demographics which don't intersect with geeks that often. Women, the very young, the elderly, non-English speakers, etc etc, were across a very real digital divide from the D&D players. You could still find advice on how to make an apple pie online, but if you did, it was because you got lucky and had a CS professor with quirky interests (for a CS professor, at any rate).

This started to change with the widespread adoption of content management systems, which took the level of computer skill for content creation down from "close to programming" to "close to using a word processor." The first very popular CMSes were blogs, and there was much triumphantalist backslapping among bloggers that blogging was democratizing the Internet. You could be blogging in your pajamas and still take on the New York Times, or so the argument went.

Ability to use a word processor is more widely spread among the population than webmastering skills, but it is still a far cry from universal. Blogging caught on primarily with professional communicators: professors, journalists, and other folks who had long been using skill with the printed word and perceived authority with pre-existing audiences. Concurrent with this, there was an explosion of content creation aimed at the concerns of well-educated, middle-class American white urban professionals. Politics, financial advice, education, religion, international news: covered, covered, covered, both by established media and publishing interests moving online and by the new media (rather like the old media, except with orders of magnitude lower capital requirements). Content was now a democracy, in the same sense that America after the Revolution was a democracy: white property owners could be reasonably assured of having their interests represented.

There still existed massive demand -- unharvested attention -- for content outside the early adopters of the Internet. Larger scale online publishers began to go after the head of the demand distribution, and hobbyist sites continued to publish things like apple pie recipes, often with a quantum leap in presentational quality from just a few years previously. Google AdWords was one of the primary lubricants for making this happen -- a hobbyist site dominating a niche like e.g. apple pies could suddenly generate non-trivial amounts of money for the site owner, largely by taking transaction costs about negotiating advertising sales out of the equation. This also allowed Google to monetize its own attention surplus better, because sending a searcher to a site with AdSense on it gives them a second chance at getting paid for a click. AdSense has generated roughly a third of Google's revenue for the last several years.

The Industrialization Of Content Production

With technology continuing to bring down barriers to creating content and business model optimization like AdWords improving the opportunity to monetize attention, it was virtually inevitable that eventually the supply and demand curves would cross. They long since had for high-value verticals like e.g. mortgages, where huge transaction volumes, high margins, gigantic advertising spends, and liquid affiliate/lead gen markets have long subsidized huge volumes of content creation. Many quite savvy Internet users were simply unaware this had happened, since one does not search for mortgages or poker every day. The Internet is a virtually uncountable multitude of attention markets, and in many of them it was more expensive to create content than the harvestable attention could justify. Those niches continued to be underserved, in the capitalist sense of the word: people would have consumed more content for them, but that content did not exist.

Then disruptive innovation happened: basically, a number of firms figured out that the combination of algorithmically predicting attention plus outsourcing content creation could let them exploit relatively small amounts of attention, in parallel, at massive scales. This innovation caused the supply and demand curves to cross for a huge number of attention markets which had not crossed before. The result: content farming at massive, massive scale.

Consider bingo cards for elementary schoolteachers, a very niche subject that happens to pay my rent. Attention exists for it: bingo has long been used in American classrooms to review vocabulary across a variety of subjects. As teachers and parents gradually started using the Internet and using Google, their attention about bingo -- a tiny, tiny sliver of the massive river of attention Google controls -- became up for grabs. Some flowed to hobbyist sites like my own, some flowed to larger publishers like the NYT's About.com unit, and some was simply poorly served. Teachers typed queries into Google and got garbage results which were not responsive.

I have advertised on Google's AdWords Content Network for years, and for the last four years I've been essentially willing to buy as much traffic as Google cares to sell me for a range of quality below a given price. This makes my AdWords stats a proxy for who is getting traffic for bingo-related searches. (Google controls navigation on the Internet, so the presence of traffic for near-term desires like bingo cards strongly suggests that it was searched for. Check your Analytics if you don't believe me.)

My market has massive seasonal changes in attention, so let's look at consistent month-long slices of it, compared year-to-year. Here's a tale of four Februaries.

  • In 2008, my AdWords spend was dominated by legacy Internet publishers like About.com, niche publishers in education, and hobbyist sites. Total spend was about $370, of which About captured almost $70 (~19%).
  • In 2009, hobbyist sites and niche publishers decline with the ascendancy of a new publisher called Kaboose, an early iteration of a content farm, focused on topics of interest to women (including, e.g., bingo). Total spend was about $560, of which Kaboose captured almost $160 (a whopping 29%), more than quintupling their performance from 2008. Or, to put it another way, more than half of increase in the size of this small attention market can be attributed to one publisher. 2009 also sees a new site in my top 10: a minor player called eHow run by an obscure firm Demand Media.
  • In 2010, spend again increases (to $640), and the top positions are dominated by content farms and ezinearticles, a legacy crowdsourced content farm. Kaboose loses share to new content farm entrants, and eHow has comparatively modest 50% year over year growth. Content farms now control over a third of this attention market.
  • In 2011, spend again increases (to $920 -- nearly 50% year over year growth), and content farms dominate the attention market. eHow has improve its execution again, to the point where they singlehandedly capture $150 in ads, quintupling performance from a year before. (Yep, their revenue is now ten times what it was in 2009.)

The Microeconomics Of Content Farming

Why did content farming capture so much of the attention economy so quickly? Basically, once the process for creating content very responsive to a single search term was repeatable, it could be replicated down the long-tail very, very quickly, in response to market signals such as e.g. successful pages in related searches. My business has long had a page about Valentine's Day bingo cards because I know, being a publisher in the niche, that they're very valuable -- there exists a substantial amount of attention which will be paid to Valentine's Day bingo every February. Do you think Valentine's Day bingo cards is a tiny niche, on Internet scales? eHow has over thirty pages targeting variants on this top -- thirty slices of a fraction of a tiny niche which were worth individualized effort to target. Some representative titles:

  • Church Valentine's Party Games
  • Make Valentine Bingo Cards
  • Classroom Valentine's Day Party Games
  • Valentine's Math Games
  • Christian Valentine's Games
  • Christian Adult Valentine's Games
  • Valentine's Party Games For Older Kids
  • etc, etc, etc, etc

Zooming in on the performance of just one of these pages, about Valentine's bingo for churches, I paid $9 for ads on it in February 2011. If we make the unreasonably pessimistic assumption that it never makes money except in February, and that the remnant image advertising is basically a wash (not true, given the amount that Groupon and online games throw around at monetizing it), this suggests that the four text ads on the page probably generated on the order of $30 in revenue. Google's 68% revenue share means that Demand Media got about $20 in revenue from this page... in 2011 alone.

Content farms are targeting evergreen content, though: Valentine's Day is going to happen in 2012, and there will still exist churches who want to play bingo on it. Will revenue from this page go to zero? That is highly unlikely, because this page wasn't written in 2011 -- it was written in 2010, when I paid $1 for ads in it (implying about $2 in revenue). Due to changes in the search environment and Demand Media's increasing sophistication with leveraging internal traffic, it got nine times more valuable at no marginal cost in the course of a single year.

Content farms operate on a portfolio strategy: the pieces of content which succeed, like that page, subsidize the pieces of content which don't. As long as the average revenue portfolio-wide exceeds cost of content production, one should expect the content farms to pour capital into content production and scale it to the moon. The portfolio strategy appears to be winning, judging by eHow's meteoric rise in revenue and the demonstrated ability for content farms to choke out non-farming content sources. eHow alone showed my ads on five times as many pages in 2011 as in 2010.

And why wouldn't they? The unit economics of content farming are stunningly attractive. Demand Media pays on the order of $10 to have the 312 words on that page written and edited. If Wall Street could design an equity which cost $10 and paid $2 per share in 2010, $20 per share in 2011, and an unknown but positive amount thereafter, all other investment classes would be virtually obsolete. The only problem is systemic risks.

The only thing that can reverse this is content getting more expensive to create or attention getting scarcer (or harder to monetize) for these markets.

There is more attention to monetize: It is possible that Internet use will decline in the future, but I will offer excellent odds to anyone who wishes to bet that: Kansas schoolmarms in the elementary bingo market have quite a ways to go before they catch up to the average reader of this blog in online consumption, which predicts a large aggregate increase in attention harvestable on the Internet and even larger proportional increases to the attention markets they care about.

Google and advertisers increasing cost of attention: Ignoring huge sources of attention of dubious worth, like ads displayed next to Facebook games, an AdSense ad displayed to someone 2 seconds after they type in a query into Google is, essentially, a search ad.

Read that again, because it is important.

This means that content farms are essentially in the search ad monetization business -- i.e. the most profitable business in the history of the Internet. Search ads monetize extraordinarily well because in addition to capturing user attention they come with user intent. This makes them orders of magnitude more valuable than the old banner display networks (which users quickly become blind to), sidebar ads next to Farmville or pictures of that cute girl from chemistry class, and the like. Content farms preserve search intent because the laser targetting combination of their one-topic pages and AdSense means that the AdSense ads are guaranteed to be responsive content to the search and, give that everything else on the page was written by a content farm, the ads are the best content on the page.

Sure, farms cede a large portion of the reach of search to actual search engines, since they can't rank for head queries, but even 5% of Google's market cap would be nothing to sneeze at. Google has incentives to help them rather than competing with them. Meanwhile, any market with competitors will tend to drive the cost of ads up until they have expended all of their margin on the sale. For a high-margin category like software, if my competitor is willing to pay 51% of his sale price to generate one marginal sale (when you back it out to cost-per-click prices), I'm willing to bid 51%. The equilibrium outcome is that my advertising costs increase over time while my ROI decreases, but it remains profitable and I'd be a fool not to do it. Google and their publishing partners win and win big.

This Is Old News. Google Fixed Content Farming... Right?

Back in late February 2011, Google rolled out the Panda update, which was widely perceived to be aimed at content farms. What actually happened was that it separated Content Farming 1.0 from Content Farming 2.0 -- earlier entrants like ezinearticles and Mahalo (and a raft of sites you've never heard about) lost out to better executing farms, including eHow.

For example, instead of comparing Februaries like we did earlier, let's see the progression of Marches in the bingo niche. Largely due to the absence of Valentine's Day, March consistently has less attention available than February: aside from an anomalous 2008 (long story with short moral: don't bork your AdWords code), spends fell 28% in 2009 and 17% in 2010. The decline was much more pronounced in 2011, possibly attributable to Panda reshaping the attention economy landscape: it jumped to 37%.

However, the performance of individual publishers was mixed:

  1. eHow (Demand Media) declined only 18%. This looks virtually in line with historical seasonal trends (growth in 2010 was so fast they were actually flat over the interval, i.e. growing much faster than market). Their performance in March 2011 (historically a "bad" month for bingo attention) crushed their performance in February 2009 (historically a "great" month for bingo attention). One could be excused for believing eHow was not net-affected by Panda
  2. LoveToKnow got annihilated -- spend decreased 71%. (The comparable decrease in 2010 was only 25%.)
  3. ezinearticles got annihilated -- spend decreased 68%. (The comparable decrease in 2010 was only ~10%.)
  4. About.com was severely affected -- spend decreased about 46%. (The comparable decrease in 2010 was, again, lower -- only 21%.)

Summed over all the content farmers, Panda appears to have picked a winner with regards to this slice of the attention economy: eHow.

I had been wistfully hoping that when the content farms got crushed that my site, which competes with them for many queries, would pick up some of the redistributed attention. If this happened, it has been too minor to notice in my Analytics stats -- my organic searches from Google look roughly in line with where I would expect them to be absent Panda. The big winner and the big losers appear to be concentrated among farmers, with fairly minor spillover to the rest of this sliver of the attention economy. This makes sense to me, in a way -- I simply don't have a page which is more responsive to the need for church Valentine's bingo activities than eHow's does. I believe my pages are far and away better than eHow's -- my pages about making bingo cards will actually let you make bingo cards -- but reasonable people could disagree on whether that is more important than capturing all parts of the user's intent, including the "specifically for churches" bit of it. Outside of my narrow slice of the online experience, a Big Publisher advocacy group estimates that the Panda update redistributed $1 billion in advertising revenue, which is nothing to sneeze at. However, with the Content Network generating over $20 billion in annual ad sales, $1 billion looks less like a fundamental shift and more like repartitioning scraps left to the losers.

Did Panda Kill Farming?

Only the economics can kill farming, and it does not appear that Panda meaningfully changes the microeconomics of content farms. If you can sell $40 of ads against a $10 page prior to Panda, and after Panda you can only sell $20 of ads, well, farm on. The model scales to the moon as long as the portfolio is even marginally profitable. The losing farms will also be incentivized to go back to the drawing board and reevaluate where they place their content bets: perhaps it is no longer lucrative enough for them to go after certain micro-markets, like elementary school bingo cards (or like the bottom half of elementary school bingo cards), but their more valuable markets are probably still stupidly profitable. Those will get more competitive as they redeploy their content creation resources going forward, assuming they're capable at executing on that.

Demand Media, on the other hand, is grinning like the cat that just caught the canary. Not only is their core business proposition virtually unaffected, in spite of the worst nightmare of their business model (Google coming down on it like the fist of an angry god) coming true, their unit economics down the tail just got radically better. Going forward, they can expect less competition in the less lucrative markets, allowing them to capture larger fragments of the attention available in those markets, and proportionally higher revenues.

The Future: Outfarming The Farmers?

A frequent theme of dystopian science fiction is that man-machine hybrids outcompete the human race. Algorithmic/freelancer hybrids, like content farms, are pretty much there, for a large and increasing portion of the content tail. This is going to get exacerbated by changes in content production and consumption, such as the rise of video (which has orders of magnitude higher production costs than text) and decline of hobbyist content creation. In 2006, my business had significant competition for keywords from individual teachers' sites, where Mrs. Smith decided (back in 1996) to put up a web page to try out this new Internet thing on her computer. In 2021, there will be many less sites created by Mrs. Smiths, because Mrs. Smith in 2011 now has an iPad to watch her Khan Academy videos on, and the iPad is virtually useless for creating websites. I'm already seeing anecdotal behavioral changes in my customers ("Say, how do I hook a printer up to an iPad so I can make my cards from there? I hate turning on the computer -- I think it has a virus or something, and it is slow."), and ordinarily they're quite behind the curve.

Additionally, while Mrs. Smith had sufficient dedication to the niche to target the most common activities, she never made more than 5 or so pages about bingo. I have about a thousand bingo activities, created with focused application of custom software by freelancers. The content farms are making me look practically lazy with their scale of publication.

This suggests an obvious route for improvement for me: if it is stupidly profitable for me to pay $200 to Google so that it can pay $130 to eHow so that it can pay $50 to freelance writers to create 5 pages, why don't I just take the $200 and pay freelance writers to write those same 5 pages... and then 15 more? The only thing which has stopped me from doing it so far is concern about polluting the Internet. But the economic attraction of doing it is undeniable. If the choice is a user getting their bingo content from an anonymous freelancer at eHow working through their queue of 400 articles for the week or getting it from someone who is only employed to write bingo articles, shouldn't they get it from me?

This dilemma, repeated a thousand times across a thousand markets, is going to create the Internet of 2020. Break out your straw hats, folks: we are all going to be farming or, at best, a step removed from farming by paying intermediaries (Google and the farms) to do our farming for us. The main distinction is going to be between successful execution of farming strategies (like eHow) and poor execution of farming strategies (like their competitors who recently got whacked).

Demand Media is already shopping out their business model as a service: since newspapers and other legacy publishers are a) dying but b) scare Google (because they can cause Google to have bad PR, which might result in government regulation, which is Google's sole competitive risk), Demand Media would love newspapers to be the front man for their farmed content. That, or parallel arrangements, is going to be almost irresistible to anyone with sufficient signals of trust to rank for arbitrary longtail content in their niche. I mean, "Create a repeatable process to create content of a known level of quality, throw money at the process to scale it, then sell ads against the result" is the entire newspaper business model! Content farming just takes out the sucky bits like "own a multi-billion-dollar distribution network for dead trees" and "write articles which are relevant to a few hundred people at an amortized cost of over $1,000 per article." (It is an open secret that the most lucrative ads in a newspaper aren't around the news, but are in sections like Style and Travel. It does not take Pulitzer Prize-winning journalism to write articles on this season's hottest shade of fuchsia or compelling reasons to go to Cancun. "Real" news has always been a loss leader to sell advertising against their other content. If they can create ten times as much fluff at a tenth of the cost, why not? And if they can... do they need the "real" news in the first place?)

Can Google just tighten the screws with another Son of Panda update? That is unlikely to work unless they repeal the laws of economics: farming happens on every topic for which the supply and demand curve crosses. Slashing content farm's ability to rank across the board by 40% just makes a fraction of the content space monetarily unattractive to them, but the content space is virtually infinite and the ability to monetize attention is increasing all the time. If Groupon will pay for remnant inventory on a page about How To Pick Your Nose, who will compete for that attention except a content farm?

Is that the Internet I want? Probably not. But then again, I'm privileged -- as a geek, my interests in content will always be well represented on the Internet, even without monetary incentives to create it. People will go to StackOverflow to answer my questions before I've even asked them, they'll create Starcraft XII walk-through videos, they'll even write software, all without seeing a penny for it. The experience for less privileged folks, though, demonstrably sucked at the dawn of the Internet, and it is not obvious to me that removing most of the growth in content responsive to their needs is a net win for them. We might see an Internet where the content-rich win and everybody else gets farming.

Like I said... dystopian sci-fi.

Patrick McKenzie runs a small software business. When not blogging or taking over the worldwide printable bingo cards market, he is working on his new venture, Appointment Reminder.

Google Wants to Act Like a Start Up

I just saw this Google snippet while trying to fine one of our old posts and it was *so* awful that I had to share it.

This is an area where Bing was out in front of Google & used a more refined strategy for years now before Google started playing catch up last fall.

Google ignored our page title, ignored our on-page header, and then use the 'comments' count as the lead in the clickable link. Then they follow it with the site's homepage page title. The problem here is if the eye is scanning the results for a discriminating factor to re-locate a vital piece of information, there is no discrimination factor, nothing memorable stands out. Luckily we are not using breadcrumbs & that post at least had a somewhat memorable page URL, otherwise I would not have been able to find it.

For what it is worth, the search I was doing didn't have the words comments in it & Google just flat out missed on this one. Given that some huge % of the web's pages has the word "comments" on it (according to the number of search results returned for "comments" it is about 1/6th as popular online as the word "the") one might think that they could have programmed their page title modification feature to never select 'comments' as the lead.

Google has also been using link anchor text sometimes with this new feature, so it may be a brutal way to Google-bomb someone. It is sure be fun when the political bloggers give it a play. ;)

But just like the relevancy algorithms these days, it seems like this is one more feature where Google ships & then leaves it up to the SEOs to tell them what they did wrong. ;)

Google Throws the Book at Competitors

You can learn a lot about how search has improved over the years by reading Matt Cutts. Recently he highlighted how search was irrelevant in the past due to a lack of diversity:

Seven of the top 10 results all came from one domain, and the urls look a little… well, let’s say fishy. In 1999 and early 2000, search engines would often return 50 results from the same domain in the search results. One nice change that Google introduced in February 2000 was “host crowding,” which only showed two results from each hostname. ... Suddenly, Google’s search results were much cleaner and more diverse! It was a really nice win–we even got email fan letters.

Thanks to those kinds of improvements, in 2011 we never have to look at search results like this.*

* And by never, I mean, unless the results are linking to fraternal Google pages, in which case, game on!

Why should Google result crowding not apply to Google.com? Sure they can say those books are from different authors, but many websites are ran by organizations with multiple authors. Some websites are even built through the partnerships of multiple different business organizations. Who knows, maybe some searchers are uncomfortable with every other listing being an out of context book highlight.

In the past I have been called cynical for highlighting stuff like the following image

I saw it as part of a trend toward home cooking promotions. And I still view it that way. The above books promotion is simply further proof of concept.

Outside of...

  • Youtube
  • other Google owned and operated sites
  • a branded website ranking for its own brand

Can you show me *any* occurrence of a result where a site is listed 5 times in the search result? Bonus points if you can find it where the 5 times are not grouped into 1 bunch via result crowding.

Other than a home cooking override, how is it possible that this problem fixed years ago suddenly re-appears?

As a thought experiment, ask yourself if that Google ranking accident would happen if the content archive being served up was promoting media hosted on Microsoft servers.

A friend of mine summed it up nicely with:

well, it's not everyday you see that kind of power and the fact that other sites aren't afforded the same opportunity makes me think that they are being anti-competitive. Google literally wrote the book (ok scraped it) on anti-competitive practices.

Majestic SEO Fresh Index

Majestic SEO has long had great link data, but their biggest issue has been usability. They sorta built with the approach of "let's give them everything" as a default, and then allowed advanced filtering to be done over the top to generate custom reports.

For advanced users this type of set up is ideal, because you are able to slice and dice it in many ways on your own terms. It allows you to spot nepotistic networks, pinpoint strategies quickly, and generally just give you a good look at what is going on in ways that wouldn't be able to do if you couldn't get all the data in a table. There are so many valuable edge case uses that can't practically be put in a single interface while keeping usability high for the average use.

But for people newer to the SEO game & those looking for a quick source of data the level of options can be a bit overwhelming when compared against something like Open Site Explorer. A parallel analogy would be that when I want to spot check rankings real quick I rely on our rank checker, but if you want to have a variety of in-depth historical views then something like Advanced Web Ranking can be a quite helpful tool.

In an attempt to improve the "at a glance" style functionality Majestic SEO announced their new site explorer, which puts more data at your fingertips without requiring you to open up an Excel spreadsheet:

How much can you use the Majestic Site Explorer?
The system is designed for silver users and above. Silver subscribers can query upto 10 different domains an HOUR. Gold subscribers can query upto 30 different domains an hour and Platinum subscribers can query upto 100 different domains an hour. All levels are subject to fair use terms.

These allow you to view data on a sitewide basis, at the subdomain level, or drill down to individual pages.

Here is an example of a site level report

and if you wanted data down to the URL level, here is an overview of a top few links (note that the report goes on for numerous pages with data)

This update helped Majestic SEO close the gap a bit with Open Site Explorer, but a couple more things they may want to consider doing are

  • adding result crowding / limit results to x per domain
  • allowing you to filter out internal link data

Those features are available via their advanced reports, but making it easier to do some of that stuff in the "at a glance" interface would allow Majestic SEO to provide as a best in breed solution for both the "at a glance" function and the "in-depth deep research" options.

Majestic SEO also announced their new fresh index, which allows you to view fresh link data as recently as within the past day. It doesn't require waiting for a monthly update or such, but offers link data right away. To help spread the word & give everyone a chance to see some of the new features they gave us free discount voucher codes to give out to get a 20% discount on your first month at any level.

If you have any questions about how Majestic SEO works you can sign up & register your own site, which allows you to access many of their features free. As a comparison SEOmoz (which offers Open Site Explorer) is also running a free 1-month trial right now.

Love & Farming vs Exploitation: Who's Winning?

Value Systems

Many broken belief systems that exist do so because of a misinformed understanding of how the world works through naive idealism, with various special interests paying to syndicate misinformation that coincides with their current business model to foster culturally constructed ignorance - agnotology.

It is not a bubble. This time is different. The internet changes everything

And then of course we had "Real estate always goes up!"

Who was behind that lie? The bankers, the mortgage brokers, the Realtors, bond raters, hedge funds, construction companies, media running real estate ads, local government tax revenues, current home owners who kept seeing their "savings" go up while doing nothing. Some of those people did not intentionally aim to be deceitful, they just believed a convenient lie that fit with their worldview.

"It is difficult to get a man to understand something when his job depends on not understanding it" - Upton Sinclair.

Fraud vs the Stuff Bankers do

And so the bubble grew until one day the fraud was so integrated into society that there was simply nobody left to sell to.

Then the bottom fell out.

The rule of law was SELECTIVELY & arbitrarily enforced against a few, even while companies that made sworn statements admitting to doing literally millions of times more damage were not penalized, but rather bailed out / promoted.

“In mid-2006, I discovered that over 60 percent of these mortgages purchased and sold were defective,” [Citibank's] Bowen testified on April 7 before the Financial Crisis Inquiry Commission created by Congress. “Defective mortgages increased during 2007 to over 80 percent of production.”

The rule of law only applies to those who lack the resources needed to subvert it. Socialism for the rich, capitalism for the rest!

Wachovia was a strong brand. A true pioneer and market leader in the drug money game, which funneled over 1/3 TRILLION Dollars onto the hands of drug dealers. For the crime they got a slap on the wrist. there was no bonus clawbacks. There was no jailtime. There was no honest attempt at the rule of law.

Online, Just Like Offline

The same is true online. Those who exhibit desirable characteristics are promoted & those who do not fit such a frame are left to fight amongst other losers in the market.

"we actually came up with a classifier to say, okay, IRS or Wikipedia or New York Times is over on this side, and the low-quality sites are over on this side." - Matt Cutts

Create an Itch

For a marketer to say what is old and steady and boring is effective is not a way to be perceived as relevant (that old coot is still stuck in '97!)

Being grounded is not a way to get positive headlines. Saying that the web is becoming just like the fraud laced offline world would be considered in poor taste. You have to sell something new...to try to push to inspire, achieve, gain hope, etc.

If you manufacture evidence that your LinkedIn votes are directly tied to better Google rankings then outsiders who are unaware of the workings of your industry may syndicate that misinformation. Even if you run a public experiment that fails it still shows you are trying new things (are cutting edge), and is a low cost branding exercise. Just like how MLM folks say you can get rich by using the same system they used to get rich. Everyone wants to sell a life worth living, even if they are not living that life, but rather sentenced to life in prison.

Pushing the Boundaries

Most profitable belief systems sell into an existing worldview but with a new hook on it. Most new marketing approaches are all about pushing the boundaries of what exists, probing to find the edges. Some people do it on the legal front, others probe on the ethical front, and yet others are just more creative & try to win by using technologies in unique ways. If you never fall off a cliff and never have any hate spewed your way then you are likely a bland marketer who hasn't done very much.

Google is in the press almost every week, aiming to stretch the boundaries on trademark, copyright, privacy, and so on.

I was lucky enough to chat with Larry one-to- one about his expectations for Google back in 2002. He laid out far-reaching views that had nothing to do with short-term revenue goals, but raised questions about how Google would anticipate the day sensors and memory became so cheap that individuals would record every moment of their lives. He wondered how Google could become like a better version of the RIAA - not just a mediator of digital music licensing - but a marketplace for fair distribution of all forms of digitized content.

Google is seen as an amazing company that does a limitless amount of good for the world. Yet the are up for anti-trust review and carry ad categories for "get rich quick." Google massages how they are viewed. Anytime something bad happens to their brand you can count on a new invention or an in-depth story of a rouge spammer getting torched by "justice."

For a company that is so good at manipulating outside rules & guidelines, they really lean hard on the arbitrary guidelines they create.

And they are willing to buy websites that violate their own guidelines. And they are not against running custom advertorials.

Foundational Marketing vs Public Relations Spin

The web is constantly shifting. Mailing lists, email newsletters, blogs, wikis, Facebook, Twitter, Color, etc. Most of the core infrastructural stuff is boring. But it is essential. If you don't understand email marketing or newsletters you can't create Groupon.

It is the new stuff with some sort of twist that earns the ink, which drives the perceived value, which earns the ink, which builds the actual value. But most people can't tell the difference between real innovation and public relations fluff. And so after a series of failures and burning millions of Dollars of capital it is time to pivot again. Anything to be seen as new and/or relevant.

If you manufacture evidence that your new strategy is better than Google then outsiders who are unaware of the workings of your industry may syndicate that misinformation. Even if you run a public experiment that fails it still shows you are trying new things (are cutting edge), and is a low cost branding exercise.

Sounds familiar, right?

History keeps repeating itself.

Algorithmic Fallout vs Spam

The perfect algorithm is something that does not exist.

Every choice has winners and losers. No matter what happens to the network & how the algorithms evolve people will find ways to exploit them. Many of Google's biggest holes were caused by Google patching old holes.

Which is precisely why Google leans so hard on public relations & shaping market behavior.

It is not the fault of the search engineer when something goes astray, but rather an evil exploitative spammer (even when Google's AdSense is the revenue engine driving the project).

Clean Your ____ Up!

Thinking back to the content farm update (which was never called the content farm update, because it impacted a wide array of websites) the main message that came out of it is that "Google can determine content quality" and "you better increase your content quality." Webmasters who heard that message were stuck in a tough situation if they had hundreds of thousands or millions of pages indexed in Google. How exactly do you *profitably* increase the quality of millions of pages, even while your site is torched to the ground, revenues are off over 50%, and the timetable + certainty for the solution are both unknowns? In many cases it would be cheaper to start from scratch than to repair the broken mess & deal with all the known unknowns.

Based on Google's advice many webmasters decided that as part of their strategy they would improve the quality of some of their best pages & then have a look at some of their worst content sections and try to block and/or remove them from Google. That sounds pretty logical! In response to that overly-logical approach to problem solving, Matt Cutts wrote the following:

What I would not recommend is sending tons (as in, thousands or even tens of thousands) of individual url removal requests to the url removal tool. And I would definitely not recommend making lots (as in, dozens or even more) of Webmaster Central accounts just to remove your own urls. If we see that happening to a point that we consider excessive or abusive, we reserve the right to look at those requests and responding by e.g. broadening, cancelling, or narrowing the requests.

So here you are trying to comply with Google's latest algorithmic approach (after they already torched your website once) and they have to give you another "or else."

Why The SEO Consultant Will NEVER Go Away

It would be nice to know what pages Google thinks are of low quality, but they don't say. It would be nice to know what pages are indexed in Google, but even official data given in Google Webmaster Tools varies widely over time, let alone the data which is shared publicly.

Further, some sites, like forums, are hard to edit to please Google without potentially destroying the flow of the community and enraging the community. Should sites have to delete or de-index their water cooler area because of Google?

What about the pages that GoogleBot arbitrarily creates by putting keywords into search boxes and generating pages that the site owner may not even know are indexed?

The reason so many webmasters are forced to rely on external search advice is that Google's desire to not be manipulated is so strong that they frequently appear dishonest & not worthy of trust. They speak vaguely, distort, and change the numbers as needed to fit the vision. Saying "in an ideal world" doesn't make that ideal world appear. And people don't trust folks like Donald Rumsfeld - at least smart people don't.

And that is why the SEO market will never die.

Corporatocracy

As for the web, it is still teething. We are most alike in the areas where we are vulgar & we are most unique the areas in which we are refined. Ultimately what happens as Google becomes more corporate is that Google becomes a boring shopping mall.

The search world loses love & farmers. But unfortunately it was the wrong kind of farmers, as eHow lives on.

Rank Tracker Feedback

With the old thread having nearly 300 comments I thought it was good time to start a new thread. Have feedback and/or questions about our Firefox Rank Checker extension? If so, ask below.

Reddit Distilled Virante

Humility

When I graduated high school one of my teachers gave me a card stating how they appreciated my humility. My mom read that and was proud, and I felt a bit embarrassed because I didn't know what the word meant and had to ask. It turns out it is easy to confuse ignorance for humility! :D

Marketing is chuck full of humbling teachable moments. One of the most important concepts is the importance of humility. When Lady Gaga spoke at Google she came off as being totally humble. I don't care for her music, but have a lot more respect for that sort of marketing than the "rapper who has more money than Bernake" sort of approach.

Exploiter Dirtbags

A lot of well known online marketers are the exact opposite of Lady Gaga's approach: anything to be heard & and I am the best, etc. Some people are into that approach, while others find it distasteful. Of course in any market there will be competition with winners and losers as SEO is a zero sum game. But some folks work to build value & monetize, while others aim to exploit & scam.

Because there are eploiter dirtbags working the market, you have to pay attention to what the market is saying about your stuff in real time. Even if you pour 20 hours into creating something that is useful, relevant, engaging, interesting, etc. some people will think it is spam just because it uses a format that spammers have exploited. Notice how in spite of our collateral damage piece being fairly well received across the web some spots that referenced it immediately raised the "spam" concern simply because it is an infographic!

Every Profitable Company is in the Gray Area

Part of the reason you need to track viral stuff in real-time is because people are tuned to think that anything in online marketing that is successful has some layer of deception to it.

In spite of how many marketers love to wear the white hat label, the truth is that almost anyone who is profitable operates somewhere in the gray area. Google has something like $40 billion in the bank, and yet they still have an AdSense category for "get rich quick." They proudly claim how they took down the get rich quick scammers that were trading on the Google brand, but they still have an AdSense category for "get rich quick."

White Hat? Probably a Liar

The problem I have with those who love the white hat label is that many people who claim to adhere to algorithmic best practices are often willing to crap on real people to get ahead. Jason Calacanis claimed to be white hat precisely because he was aware of how dirty and exploitative his Mahalo junk was. You can't really get away with flagrant spamming if you call it what it actually is, so you have to preach righteous virtues while doing it.

That "scammer exploiting a loophole" approach can work for your thin affiliate site that isn't tied to your name & brand, but if you are using that sort of stuff on your client projects or on your own main brand site you build contempt in the marketplace. Which is precisely why so many SEOs were happy to see Mahalo get torched in the content farm update.

The recent "advanced" link building conference brought about 2 teachable moments on that front.

How to Breed Hate & Animosity in Your Marketplace!

Before the conference Will (from Distilled) asked me if I would be ok with him writing a post here. We have tried to be fairly neutral in the marketplace (reviewing tons of competing sites and products and whatnot), so I said sure. He handed me something that I found to be pretty offensive. To which, when he asked for a follow up, I replied:

Generally I felt that suggesting that post was sorta a smack in the face. It was like an ad inside an ad inside an ad. Ad for seminar + ad for your seo services + laundry list of links to client sites.

That you would suggest that made me feel like you think I am stupid or that you were trying to disrespect me. I didn't reply right away because I was a bit angry at the time & didn't want to respond that way. And then all that tech crap happened. Anyhow I think you are savvy and are a great SEO, but a post like that (ad in ad in ad) is better fit for say like John Chow's blog than ours ;)

He responded with how much of a fan he was of mine & that if he knew of anything cool on the news front down the road he would try to help us break it & such.

At the conference he highlighted his "appear authentic, but be driven by a script" type of approach.

Advanced!

But what he *failed* to disclose (until his brother disclosed it during the conference as part of the conference) was...

  • that they had been hired by a competing SEO site to try to outrank us for SEO tools
  • that they suggested the site they were working on to use 301 redirects to game Google
  • that the site they were working for outed our site for using 301 redirects & got it toasted in Google

In other words, where a person who is truly ahead of the market, and does something to create a competitive advantage it must be black hat spam and you should complained to Google to get it torched. Then years later when the people who claimed the technique was spam do the same damn thing it suddenly becomes clean and innovative (cutting edge advanced stuff even).

Then they want the person who was ahead of the curve to be a free conduit for spreading this trash! It is so bad that you couldn't even make this stuff up.

Consider the brand damage they did to themselves & the bad karma they earned in the marketplace with the above stupidity. If they are willing to do that sort of stuff to their own brand, would you want them working on your brand? I wouldn't.

Those who claim to be algorithmically white hat, but are fine with lying, being deceptive, failing to disclose conflicts, etc. are saying that they put the algorithm ahead of how they treat real human beings in their marketplace. It is fine to be exploitative if that is your approach, but be honest about it ... because it is dumb to do it in a way that causes damage to your brand.

Spam vs Junk Trash Garbage That People Hate

Some people also complain about domain names (a clear signal of relevancy) shouldn't count, and yet some of the same folks create software to automate spamming up public communities. Any competitive disadvantage they have is spam, any competitive advantage they have is not spam. ;)

Enter Russ Jones!

On the Virante about page it highlights that "Russ has assisted in the creation of new search marketing technologies. This includes the venerable LinkSleeve Spam Link Verification system, which currently blocks thousands of links spam messages across the web." Yet at the "advanced" link building conference he gave away software to help people spam the crap out of Reddit.

Once the people on Reddit highlighted it he was quick to backpedal, stating: "I don't openly promote spamming. If you think creating highly viral content and submitting it to a social network to let them decide if it is good is spamming, then you have been seriously misled."

Here is the deal though, if the goal of that sofware was to do ANYTHING other than spamming, then it would be promoted to the core audience (so it could reach more people) rather than hunting out old subreddits & spamming them up with links.

Yet again, advanced!

There is nothing new or advanced about that link building "technique." It is just an extension of guestbook or comment spam. The above image links to a Vimeo video which highlights what Matthew Haughey thinks of the SEO industry after he found out someone was selling an info-product on how to spam up Metafilter by dropping links in old posts. Slapping the label "advanced" on old spam techniques makes them neither new nor advanced. The clock moves in one direction. Unfortunately it is not 2003 anymore.

There are a bunch of exploitative douchebags that paint themselves as white hats while destroying the ecosystems we all must work in by undermining basic human decency principals & trust in the marketplace. I don't care if someone wants to be a spammer, but to do so and claim that you are white hat and ethical (and thus that others are somehow inferior) is garbage.

Even Ghetto Rappers Stand for Something

The most important lesson in marketing is consistency. Make promises that you can consistently deliver on.

Rappers are successful. So are folks like Thom Yorke. But they pick their markets & their approach and stick to it. Bouncing back and forth just makes a person look like a dishonest douchebag who stands for nothing.

It's not standing for much, but least the rappers have their drugs, booze and hoes.

What do these internet marketers stand for?

It seems the folks teaching "advanced" internet marketing still need a bit of work on "basic" social interactions & common sense. But I guess those are harder to sell. ;)

Google Panda Coming to a Market Near You

If you live outside the United States and were unscathed by the Panda Update, a world of hurt may await soon. Or you may be in for a pleasant surprise. It is hard to say where the chips may lay for you without looking.

Some people just had their businesses destroyed, whereas the Online Publisher Association sees a $1 billion windfall to the winning publishers.

Due to Google having multiple algorithms running right now, you can get a peak at the types of sites that were hit, and if your site is in English you can see if it would have got hit by comparing your Google.com rankings in the United States versus in foreign markets by using the Google AdWords ad preview tool.

In most foreign markets Google is not likely to be as aggressive with this type of algorithm as they are in the United States (because foreign ad markets are less liquid and there is less of a critical mass of content in some foreign markets), but I would be willing to bet that Google will be pretty aggressive with it in the UK when it rolls out.

The keywords where you will see the most significant ranking changes will be those where there is a lot of competition, as keywords with less competition generally do not have as many sites to replace them when they are whacked (since there were less people competing for the keyword). Another way to get a glimpse of the aggregate data is to look at your Google Analytics search traffic from the US and see how it has changed relative to seasonal norms. Here is a look out below example, highlighting how Google traffic dropped. ;)

What is worse, is that on most sites impacted revenue declined faster than traffic because search traffic monetizes so well & the US ad market is so much deeper than most foreign markets. Thus a site that had 50% profit margins might have just went to break even or losing money after this update. :D

When Google updates the US content farmer algorithm again (likely soon, since it has already been over a month since the update happened) it will likely roll out around other large global markets, because Google does not like running (and maintaining) 2 sets of ranking algorithms for an extended period of time, as it is more cost intensive and it helps people reverse engineer the algorithm.

Some sites that get hit may be able to quickly bounce back *if* they own a well-read tech blog and have an appropriate in with Google engineers, however most will not unless they drastically change their strategy. Almost nobody has recovered and it has been over a month since the algorithm went live. So your best bet is to plan ahead. When the tide goes out you don't want to be swimming naked. :)

Yahoo! Search Direct

What is Search Direct?

Yahoo! announced search direct, a new beta product launched in the United States which is similar to Google Instant, but extends a bit further.

It works by extending the search interface to include a layer before the results come up. The layer typically includes a left column of related keywords & a right box that can be anything from:

  • 3 top websites for that query
  • a weather forecast
  • stock information
  • the profile of a celebrity
  • other unique data sets

Here is an example of how the search box flies out

Here is an overview video from Yahoo!

Arbitrage or Helpful?

It is easy to laugh at Ask.com when thinking about the spammy end of the "answers engines" (or even Yahoo! Answers for that matter), but this search direct could range from highly useful to pretty weak depending on what Yahoo! decides to do with it. It's impact on various markets can range from trivial to significant.

What Powers Search Direct?

The ranking algorithm for Yahoo! Search Direct is different than their core results, being powered off a smaller index with its own algorithm, with a rapid refresh rate. Greg Sterling asked Yahoo!'s Shashi Seth about what drove the algorithm:

Seth told me that right now the links and content being shown in the right part of the box are the URLs that are the “most clicked” throughout the Yahoo network. He also implied that it might get more nuanced over time. And he added that rankings can change moment to moment because it’s dynamic.

That click bias has a natural preference toward promoting Yahoo! properties (since Yahoo! users like Yahoo! stuff) and promoting those who are featured on the Yahoo! network through editorial partnerships.

Greater Integration of Self Promotion

One of the benefits of Yahoo! outsourcing search is that they can now claim that they are not a search engine, which gets them around a ton of conflict issues, and allows them to aggressively self-promote without the type of scrutiny Google has come under for hard-coding their search results. Currently Yahoo! Search Direct is not yet running ads, but it is full of self-promotion. It is not a great sign for the longevity of Yahoo! Search that when you start typing almost every letter of the alphabet leads to a downstream Yahoo! product. In the past, search engines which have over-monetized have seen marketshare erode to Google. Hopefully this stuff pushes people to Bing though!

In key verticals where Yahoo! is well established the entire preview box is consumed by content from their vertical databases. See, for example, a search for LeBron James

If you are ESPN it becomes much harder to get traffic from Yahoo! Search directly given that sort of layout. If the model proves profitable enough Yahoo! can close off a lot of verticals. The key for web publishers is that Yahoo! has traditionally been horrible at integration, so the odds of them doing this in a way that monetizes more aggressively without harming Yahoo!'s search marketshare are pretty low. Having wrote that, last year Yahoo! bought Associated Content and has been pushing hard at growing their news, sports & finance verticals. If they are able to instantly tap a large share of the search market & can throw up a featured promotion for some of their key content then that will lead to lots of usage data (Microsoft has already mentioned using clickstream data to create a search signal) & social signals (like Facebook likes) that can bleed into improving the ranking of Yahoo! content in other search engines.

Custom Ad Units

The showing of a mini-search box not only gives them the potential for further self-promotion, but it also allows them to run more custom ad units that are in full focus of the end user. When you display a full search result you are offering a list of options, but premium placement ads in the preview box can allow for tighter integration of video, audio, or other custom ad units within search.

Yahoo! has already put sponsored mortgage rates table in their search results. Now if they want to do something like that they can have it own the whole of the interface, sorta like Google has done with their local results. It will also allow Yahoo! to test video results in the search results, something Google is getting into as well.

Yahoo! has also taken branded search ads one step further, with a wrap around on certain keyword queries, like eBay.

Shortcomings

Where Yahoo! Search Direct falls short, especially when compared against Google Instant is it's force of pushing a single vertical for keywords that can have many meanings. Take, for example, a search on cars. If you don't want the DVD, you are still forced to view information about the cartoon movie because a Yahoo! vertical has a match.

Another thing Yahoo! seems to be doing is force feeding a local option as the last suggested keyword, even where it is totally irrelevant. In the long run I think this would harm Yahoo! local as a true destination, but it can drive short term volume. Of course this only just launched, so it will likely become more relevant as they track how users interact with it. Currently someone is likely registering a Yahoo! local profile with Viagra in it somewhere. :D

Pages