How To Lie With Statistics

There are three kinds of lies: lies, damned lies, and statistics - Disreli

We get presented with graphs and statistics every day. "Most SEOs think keywords in the title tag is an important ranking factor." "Spending on search to rise by $10b". Ever get that feeling that what you're being presented with sounds plausible, but the conclusion just doesn't make sense?

Here are a few common ways people try to pull the wool over your eyes with statistics. Some you'll be familiar with. If you've got more, add 'em to the comments :)

1. Built In Bias

The sample data supports an obvious agenda. For example, a company is hardly likely to show a graph that shows their product has produced negative results. Try to determine the bias of the person or organisation presenting the data - "what would they want me to hear"? then ask yourself: "what data are they not showing me?"

2. The Average

The media loves to state "the average", then neglect to tell you which average they are talking about.

For example, the average house price for an area could both be 500K and 200K, depending on what type of average is being used. They could be referring to either the mean, the median or the mode. They often mix these up, depending on what conclusion they want you to reach.

3. Inadequate Sample Size

20% of web designers make over $1M. That may be true if the sample size consisted of ten highest earning people in the industry, and two people just happen to have had a great year. But what if the sample size is all those who practice web design for a living? The outcome may be somewhat different.

4. Meaningless Differences

A difference is only a difference if it makes a difference. Potential employee Jill may have an IQ of 120, and potential employee Jack may have an IQ of 118, but does that really mean anything? What if Jill has an attitude problem, and Jack is a great conversationalist? Who would be the better hire?

5. Oh My God!

Al Gore loves this one. The graph that shows some astonishing change in the status quo. The impression is one of significant movement and is meant to shock an audience.

However, if the chart appears in a different context - say, over a longer time period - the rise may not look all that unusual. You often see this in stock price quotes. You could also change the measurement into smaller units, thus making any movement in the graph look even more impressive.

6. What You Infer Is Up To You

If you can't prove what you want to prove, prove something else and pretend they are the same thing. Often used in the alternative medicine industry. They may not be able to prove that their natural products cure cancer, but they can say that the plant extract has been used by some remote tribe, and they have a proven historical low incidence of cancer.

7. Post Hoc

A study found students who smoked got lower grades. The fallacy of one thing not following the other i.e. smoking doesn't cause bad grades. Frequently, other factors are left out i.e. the students who smoked also tended to be party animals. Look out for correlations that happen by chance.

8. Data Precision

Quoting specific numbers, especially including decimals points, can look authoritative. "Real estate values up 4.95%" Why would someone be so precise if they didn't know their stuff? The numbers can be wild guesses, but accuracy gives an air of authority.

General Tips For Spotting The Lies

• Ask "who says so?" Are they likely to be biased? If experts are cited, check to see if those experts actually agree with the conclusions. Often, they do not.
• Ask "How do they know"? Is the sample size really large enough, or relevant enough, to draw conclusions?
• Look To See If They Change The Subject. Look for a change between the raw data and the conclusion. Does one follow the other? For example, more reported incidences of crime do not necessarily mean there is more crime occurring.
• Ask "Does this make sense?" - are they trying to blind you with numbers? If the conclusion just sounds wrong, look for a disconnect between the data and the conclusion

If you want to delve deeper in to How To Lie With Statistics, grab the little book of the same name. It's getting a bit dated now - it was written in 1954 - but the advice and examples are great :)