In my previous blog entry I briefly reviewed Darrell Huff’s excellent book, How to Lie with Statistics. In the closing chapter Huff summarises the lessons by explaining How to Talk Back to a Statistic. Or, in Huff’s own words, “how to look a phoney statistic in the eye and face it down”.
Not all the statistical information that you may come upon can be tested with the sureness of chemical analysis or of what goes on in an assayer’s laboratory. But you can prod the stuff with five simple questions, and by finding the answers avoid learning a remarkable lot that isn’t so.
– How to Lie with Statistics, Chapter 10, p110
The five simple questions are:
- Who Says So?
- How Does He Know?
- What’s Missing?
- Did Somebody Change the Subject?
- Does It Make Sense?
Armed with these five questions I thought it might be interesting to examine a real world example.
You might have seen recently the news report that “75% of ex-Bush officials are still unemployed”. The source of the story was the Wall Street Journal article of 21 Feb 2009: Jobs Still Elude Some Bush Ex-Officials
The jobless rate is hanging high for many of the roughly 3,000 political appointees who served President George W. Bush. Finding work has proved a far tougher task than those appointees expected …
Only 25% to 30% of ex-Bush officials seeking full-time jobs have succeeded … much, much worse than when Ronald Reagan, George H.W. Bush and Bill Clinton left the White House …
Let’s put this 75% unemployment rate of ex-Bush officials to the Duff Test.
Who Says So?
The first sleight of hand you notice about the statistic is that it hides behind what Huff calls an “OK Name”. In this case the “OK Name” is the Wall Street Journal, a well known and reputable news source. But it’s not the WSJ who actually “says so”. It’s not a piece of their own independent investigative journalism. In this case the WSJ is merely reporting on a statistic prepared by somebody else. It’s therefore worthwhile considering if this third party actually has any expertise in the areas of data collection and statistical analysis. Are they impartial? Could they be biased? Do they have a hidden agenda or ulterior motive behind presenting these figures?
I don’t want to drink from a poisoned well. So I’m going to approach this source with a healthy dose of scepticism.
How Does He Know?
How did the researchers arrive at their “estimate”? Via robust statistical sampling? Rumour mill? Reading tea leaves? On the face of it, the data look anecdotal at best. The WSJ article doesn’t go into any details. This is enough to raise a second doubt about the statistic.
What kind of error margin is there in the estimates? If the estimate was based on a sample, how big was it? How was it selected? Is it representative? When comparing ex-Bush officials with previous administrations are they comparing apples with apples in terms of such things as ages and career ambitions? Were ex-Bush officials more likely to be heading into retirement or satisfied with a bit of part time work? Not to mention the vastly different employment situation that exists right now as the U.S., and indeed the world, enters the Second Great Depression.
Did Somebody Change the Subject?
Huff warns that “when assaying a statistic, watch out for a switch somewhere between the raw figure and the conclusion. One thing is all too often reported as another.” Although it doesn’t explicitly say so, the implication behind the WSJ article is that ex-Bush officials are having a hard time finding employment because they’re ex-Bush officials. It’s fair to say that George W. Bush is regarded as being one of the worst U.S. presidents in history. Certainly he left office with some of the lowest approval ratings of all time. So it’s only natural that nobody from his administration could ever find gainful employment again. They’re hopeless and everybody hates them, right? Well, maybe. But the truth is probably far more complex. There are so many external variables in play that such a conclusion represents a leap of faith.
Does It Make Sense?
For any statistic to “make sense” it needs context. A comparison of yourself to a group, or your suburb to the country, or a trend over time are examples of data context. In my opinion any kind of real, meaningful context is missing from the statistic reported by the WSJ.
All in all I believe the figure of “75% of ex-Bush officials are unemployed”, as reported by the WSJ, fails Duff’s basic how to talk back to a statistic five-point criteria. This particular statistic is like a bad smell in an elevator. Source and purpose unknown, it hangs in the air requiring our attention. But any kind of meaningful comment is impossible. The only sensible course of action is to ignore it.