The Benefits of Using Google's Custom Search Engine

I recently installed the new business edition of Google's custom search engine. It took about 5 minutes to set up and will likely pay for itself many times over. You can use it by using the search box in the right rail. Google's custom search engine is easy to customize, fast to implement, works across multiple domains and is cheap. It only costs $100 a year for up to 5K pages and $500 a year for 50K pages. It also allows the owner to see search frequency and your most popular queries.

Rather than doing Google site specific searches, now I can see Google flavored results customized to my site right from my own domain. If I am searching for something on my own site and can't find it, the odds are pretty good that I am not going to show up in Google's global search either.

Using Google's search will likely also help you figure out how well they trust different pages on a site or a subset of sites, as it likely factors in link popularity and other off site relevancy measurements (unlike most site search services). If your site is easier to search it is easier for you and others to cite your archived content.

Using search results with the same format as Google's will also show you how compelling your documents look to customers on Google, from a searcher's perspective. Some consumers may also view your business as being more Google friendly if you use a Google site search service. Their awareness of and affinity toward Google's brand may help increase your conversions.

Local Data is Worthless (Unless You Have a Community)

Backfence died because it was made obsolete by Google's relevancy algorithms and older local community sites. The commoditization of local data is only going to get worse. iBegin Source allows you to search or browse local business information for free, or buy an entire state for $1,000. Today Google announced they are allowing people to overlay mapplets, which will likely make Google the default source for local information inside of 2 years.

Does Search Disrupt the Business World?

When Brin and Page wrote The Anatomy of a Large-Scale Hypertextual Web Search Engine they noted that advertising biases search engines toward the needs and wants of advertisers, and against the best interest of consumers.

Currently, the predominant business model for commercial search engines is advertising. The goals of the advertising business model do not always correspond to providing quality search to users. For example, in our prototype search engine one of the top results for cellular phone is "The Effect of Cellular Phone Use Upon Driver Attention", a study which explains in great detail the distractions and risk associated with conversing on a cell phone while driving. This search result came up first because of its high importance as judged by the PageRank algorithm, an approximation of citation importance on the web [Page, 98]. It is clear that a search engine which was taking money for showing cellular phone ads would have difficulty justifying the page that our system returned to its paying advertisers. For this type of reason and historical experience with other media [Bagdikian 83], we expect that advertising funded search engines will be inherently biased towards the advertisers and away from the needs of the consumers.

Not surprising then, when Google becomes a leading advertising engine, they go so far to cater to advertisers that Google's blog bashes documentaries while using that opportunity and platform to remind advertisers they can manipulate public perception by buying Google AdWords ads and Google delivered AdSense ads on content websites.

We can place text ads, video ads, and rich media ads in paid search results or in relevant websites within our ever-expanding content network. Whatever the problem, Google can act as a platform for educating the public and promoting your message.

Cory Doctorow offers an alternative solution

Another approach would be to reform the practices that Moore criticises in the film -- for example, refusing to pay for an insured individual's surgery because she didn't mention a 15-year-old yeast infection on her application; denying MRIs to patients with brain tumors; and paying medical directors bonuses for denying claims.

Google also has a health advertising page, which goes so far as to say

Yes, healthcare consumers are moving online. But there's more to the story than that. For instance, the Internet is the leading media source of health, medical, and prescription-drug information. In addition, the majority of consumers use a search engine prior to requesting a prescription drug from a doctor. The bottom line: The Internet plays a central role in the way consumers' access healthcare information.

This is the sort of stuff I was fearing when I wrote about Google's shift from direct marketing to selling brand ads. There is no care for relevancy if spreading misinformation pays more:

I can see how one might want to play up the fact that Google is admitting it can “use” the web sites in its content network to sway the public perception…which is really telling all those web masters running Google ads that they are Google pawns, that Google is not neutral morally or politically, and that Google seeks out opportunities to exploit that (for profit).

SEOs get a black eye for market manipulation, but is what Google suggests any better? Nope. It is only wrong to manipulate public perception or relevancy if you don't have enough money to pay Google directly. If you have a wallet open it up and let Google syndicate your spin.

Google Custom Search Engine

Creating a Google custom search engine looks like much more work than it actually is. Recently Google allowed webmasters to make them on the fly by seeding them with the links from any page(s).

Add together your few favorite sites, the sites listed in the relevant DMOZ categories, and the sites listed in the relevant Yahoo! Directory categories. Spend 10 minutes purging the spammy our outdated links and you have relevancy that might match Google's in many markets.

Google.com is the Web's Largest Social Network

Blogging, email, IM, and telephone make it easy to keep in touch with your real friends. Social networks are hyped, but tend to have low value traffic because they don't effectively separate signal from noise. The only people who have time for them are people hawking crap, people looking to waste time, and spam bots.

Sure Google owns Orkut, but they don't need it. Google already is a social network, and became one by targeting and partnering with the power users and influencers, but few people think of them as a social network. If you are a webmaster there is a good chance that Google delivers your ads, controls your feed, is your analytics provider, and is a leading source of traffic to your site. News sites and other trusted editorial partners are rewarded with high Google rankings, and become addicted to Google's traffic the way LookSmart was addicted to Microsoft's traffic a few years ago.

On top of having touch-points with most webmasters, Google also has touch-points with mass information consumers. Google's Toolbar gives them web history, which knows where you have been and how long you were there. Google Reader is the most popular feed reader service on the web. They know what voices you trust and subscribe to, and allow you to share your subscriptions or favorite items with friends.

Google's personalized homepage is also the most functional, customizable, and useful service by any of the major portals. Google gadgets are easy to create and syndicate, and Google allows you to publish gadgets or tabs with anyone.

They go so far as recommending information via the interesting items for you gadget, a recommendations tab on customized homepages, and toolbar recommendations.

Google Maps allows you to link to any location, adds data layers from trusted third parties, and accepts public comments on maps.

YouTube makes it easy to upload and syndicate videos, and now those are highly ranked in Google's search results. Look at YouTube's TestTube to see the community features they are planning.

Google offers an auto-complete feature, which makes it easy to refine your search based on the searches of others. They also show related sites and hot searches, and let you note sites you like via Google Notebook. If you don't find their results entirely relevant they make it easy to create a custom search engine using their data.

Keep track of time and your projects using Google Calendar and Google Docs.

Sure Google has Orkut, but they don't need it. By default Google keeps hold of your attention so they can use it to improve their relevancy algorithms and ad targeting. Tie in search personalization with automated ad optimization and their information sharing tools and it is easy to think of Google as the web's largest social network, though most people don't think of them that way, largely because they put information at the center of it, and are so good at separating signal from noise.

Google's Move Away From Direct Marketing to Selling Branded Ads

Ad Relevancy & Quality Scores

Google has again and again touted the value of their targeted marketing, but most of the fortune 500 ad dollars are not spent on targeted marketing. A couple weeks ago in a WebmasterWorld thread many advertisers complained about getting killed by another quality score update.

What quality score actually means probably comes down to one of two things

  • your site is a thin affiliate site or something else they once needed to fill a market niche but now is viewed as noise

  • you have not created enough organic value and/or have not yet spent enough money building your brand

Google Hates Most Affiliate Websites

Some key quotes from the WMW thread...these two show the trend against affiliate sites in general

Too many outgoing affiliate links and you are toasted

So on my basic two types of sites, when I send the visitor to another domain to buy, I'm getting severly penalized ( a new affiliate "penalty"), but if I have a another party's lead form on my domain, I didn't get hit.

and this one shows that the change is not a short term one

A Google Adwords customer service rep said that they do not systematically target affiliates as a whole, nor sites with affiliate links. But, she said they are taking more steps with each landing page tweak to weed out sites that do not add a certain level of "value" to their visitors (as other posters to this thread have mentioned). She wouldn't tell me if this "value" is human-determined or algo-determined, again saying that she didn't know.

If your site is not the type of site they would white-list in the organic results eventually they are going to look to dispose of your position in the ads as well. As soon as enough brand advertisers find your space you are no longer needed. Thanks for sharing the keyword data needed to tell the brands what to bid on, and best of luck getting traffic from somewhere else.

If you want to see where paid search quality filtering is headed, look at how the organic algorithms have changed. Nothing better to glimpse the future of PPC than to read the documents about how they expect humans to rate organic search results.

Expanding the Role of Brand Related Advertisements

In that same WMW thread Skibum posed the following question

Why attack long time advertisers regardless of their business model who are providing consumers with what they are looking for while using broad match to show more ads triggered by keywords they were not intended to run on?

I recently saw a Dollar rent a car ad at the #1 ad position for Forex, which is not a cheap keyword.

Google Expanded Broad Match Going too Far.

As the day passed Google's CTR numbers showed they expanded that ad out too far and they made that ad less broad. They can automate spreading out brand ads too far, and then pull them back if the relevancy scores are too poor.

When it comes down to it, it is all about money. As Google commoditizes everything that is not a brand they need to collect more money from brands. The reason Google is pushing video hard is because they want to lead that ad market. It is no suprise to see Google leading in innovation in the video ad field. There is no better way to create inventory than to get it from your already established near infinite traffic stream.

The Cadillac Escalade video ads are taking the place of the textual Ford Explorer ads. Google has no brand allegence. Whoever is willing to overpay for exposure right now can buy all they want from Google.

Even when Google can show relevant ads, they still prefer to show brand ads if they think they will pay more. Consider a Michigan counties page where Google shows the following ad links.
Relevant Google AdLinks.
Those are relevant. But what ads does Google also target to that page?

Google Car Donation Ad.
A lot of car donation charities are non-profit shells wrapped around dirty high margin auction houses (just look at the $20/clicks ad pricing).

Google Drug Related AdSense Ad.
A pharmaceutical ad from a company with a patent an a marketing budget larger than their research budget.

"The most startling fact about 2002 is that the combined profits for the ten drug companies in the Fortune 500 ($35.9 billion) were more than the profits for all the other 490 businesses put together ($33.7 billion)." - Marcia Angell

Given that ad targeting, it doesn't seem that Google is so pure, does it? One of the guys at WMW said the following

You guys are AdWords arbitrageurs. Although I'm sorry that your little gravy grain went off the rails, as a Google user, I can say good riddance to your garbage Web sites. Google, and users, want actual retailers to come up top in search results for sellers of a product, not parasite Web sites linking to actual retailers.

In a few years that same guy will probably be whining about how Google destroyed his business, but just like the other websites that died, Google doesn't care about him. What they want is decent relevancy WITH as much profit as legally possible. The more they cut out middle men the bigger they can make their chunk, even if doing so hurts relevancy and result diversity.

How do they get any more efficient than automating ad targeting while turning the text link into an unmarked ad unit? And they have patents for ad targeting based on how big of a risk taker you are:

"Examples of information that could be useful, particularly in massive multiplayer online RPGs, may be the specific dialogue entered by the users while chatting or interacting with other players/characters within the game. For example, the dialogue could indicate that the player is aggressive, profane, polite, literate, illiterate, influenced by current culture or subculture, etc. Also decisions made by the players may provide more information such as whether the player is a risk taker, risk averse, aggressive, passive, intelligent, follower, leader, etc. This information may be used and analyzed in order to help select and deliver more relevant ads to users."

How can anyone else compete in the ad market?

Inside Google's Black Box

A bit slow to mention this article that came out while I was at SMX, but it is worth mentioning. The NYT ran an article about Google's relevancy algorithms titled Google Keeps Tweaking Its Search Engine, which talks about how Google reacts to relevancy problems. The article should be required reading for all SEOs and search lovers. Rand offered a great overview of the article, highlighting that Google noticed their age weighting was too heavy when Google Finance was not ranking where they thought it should, Google changes their matching algorithms to place more weight on phrase matching for some queries, and that Google has created a query deserves freshness (QDF) algorithm which determines how well to mix in new and old results.

Matt Cutts, who was quoted in the New York Times article, also commented on it on his blog, confirming that

The search-quality team makes about a half-dozen major and minor changes a week to the vast nest of mathematical formulas that power the search engine.

Google is not the only search engine heavily focused on the human elements of search. In one of Tim Mayer's slides he showed a Yahoo! search result which said something along the lines of Yahoo! employees see bad results? Report spam.

The two things Google does more aggressively than the other engines are:

Build as many quality signals as you can, such that if they ever impart intent on your sites you can get away with more dirty stuff than a spammy sounding unbranded site can.

How to Turn Google Personalized Search Results Off Without Logging Out

At SMX Matt Cutts said you can turn Google's personalized web search results off by adding &pws=0 to the URL of a search query. If you are logged into your Google Account this link shows your personalized version, while this one shows the general unpersonalized SERP.

Data Collection: Google's Biggest Competitive Advantage

Today Google bought Feedburner, which (along with AdSense, AdWords, the Toolbar, Analytics, user accounts, Google Feed Reader, Google Checkout, Youtube, etc) is yet another source of data acquisition for them.

Earlier today I posted about a small and harmless javascript that can be used to see which competing sites visitors visit before going to your site. In the second comment about it a guy named Dave nearly exploded.

While everyone is running around polluting links on the web graph (and fighting over who the spammer is and what is spam), Google is busy building something only they can build, because they are the only ones who get a free pass on collecting user data as a feature.

Google Kills Low End AdSense Arbitrage & MFA Spam

eBay recently banned search arbitrage, and Google is also trying to clean up their network. eWhisper posted that he thought the recent clean up may be an attempt to make the network look more appealing before showing advertisers what URLs their ads appear on. A neverending Webmasterworld thread highlights that many thin AdSense publishers got emails from Google stating that "our specialists found that your business model is not a good fit for the AdSense program".

The hard part is both judging intent and forcing quality. On one front Google offers a heat map showing you that your ads should go in the middle of your content, then they tell you to use AdWords to drive traffic to AdSense sites, and in their guidelines they state:

No Google ad may be placed on pages published specifically for the purpose of showing ads, whether or not the page content is relevant

In a market where virtually every business arbitrages other markets what is an improper business model? SEO is a form of arbitrage. I even had one engine call me up and ask to pay me to spam arbitraging Google's search results. That engine may even be one of Google's largest syndication partners! As ShoeMoney pointed out, many large arbitragers will continue arbitraging. Companies like Business.com or the 100s of shopping search engines are not going to get booted out of syndicating ads.

I think the big long-term distinction between what Google considers low quality and of quality is going to be brand equity. Do people visit your site from channels other than Google? Large brands get more return out of buzz marketing, while smaller businesses lean hard on search. inbound made a great post in that WMW thread about the arbitrage and MFA changes, which notes that Google is getting better at coming up with proxies for visitor value.

Based on Google's authority-centric relevancy algorithms and this move it is clear that Google wants to trust the larger businesses so they have less work to do policing the web. The way around getting forked by Google is to create something that does not need Google to exist. Create the type of site that people would link at if Google did not exist, and the type of site that they would want to advertise on directly. I have a large AdSense site that needs a re-brand and some serious work on content quality if it is going to stay viable in years to come.

For those who just got the death letter here are a few options:

  • Buy old trusted sites with a clean core and real value add. do arbitrage stuff off of them

  • Look to other related programs like YPN, Searchfeed, and Ask.com's feed.
  • If your site is of low value consider showing fewer ads short term...to make up for non arbitrage ads and to lower your risk profile.
  • Create a new corporation, open up a new AdSense account and figure out ways to get your sites approved.
  • Publish fewer and higher quality sites with fewer ad units on each page.

From a risk management perspective, I think every web publisher should own at least one real branded site, even if it offers low returns for the amount of work required to maintain it.

Pages