Some time ago I came across an article in NewScientist: Porn in the USA: Conservatives are biggest consumers.
Americans may paint themselves in increasingly bright shades of red and blue, but new research finds one thing that varies little across the nation: the liking for online pornography.
A new nationwide study of anonymised credit-card receipts from a major online adult entertainment provider finds little variation in consumption between states.
“When it comes to adult entertainment, it seems people are more the same than different,” says Benjamin Edelman at Harvard Business School.
However, there are some trends to be seen in the data. Those states that do consume the most porn tend to be more conservative and religious than states with lower levels of consumption, the study finds.
“Some of the people who are most outraged turn out to be consumers of the very things they claimed to be outraged by,” Edelman says.
As I waded into the paper, things began to get murky. Firstly, the study relied on only a single source of data, a “top-10 seller of adult entertainment”. In fairness, Edelman does admit that “… it is difficult to confirm rigorously that this seller is representative” but explains it away with “… the seller runs literally hundreds of sites offering a broad range of adult entertainment.” Crikey. That old chestnut. If your sample is large enough it cancels out any bias? This is the statistical equivalent of “if lots people tell you something it must be true”. I believe it’s worth raising a proverbial sceptical eyebrow here at the source of the data.
Secondly, a little sleight of hand. “Little variation in consumption between states” segues into “there are some trends to be seen in the data. Those states that do consume the most porn tend to be more conservative and religious than states with lower levels of consumption.” To me, the statements look mutually exclusive. Either there was little variation or there was some trends to be seen. Which is it? I do feel for the researcher who slaves away for many hours on a project, only to yield a null result. It’s never as exciting as finding that diamond in the mud. But a null result is still a null result, and it’s important to stay true to your findings. Hell, just ask Michelson and Morley.
But what concerned me the most was the decoupling of the conclusion from the results. Endelman concludes that “some of the people who are most outraged turn out to be consumers of the very things they claimed to be outraged by.” In other words conservatives are hypocrites because they actually buy more porn than those Godless, unbathed liberals.
This is an example of an Ecological Fallacy.
An ecological fallacy, often called an ecological inference fallacy, is an error in the interpretation of statistical data in an ecological study, whereby inferences about the nature of specific individuals are based solely upon aggregate statistics collected for the group to which those individuals belong.
Put simply, let’s say I went to a statistics professor with these results:
State X, 1000 people, 90% conservative, 6 porn subscribers.
State Y, 1000 people, 90% liberal, 3 porn subscribers.
If my conclusion from those data was that conservative people buy more porn than liberal people, the statistics professor would simply laugh at me and walk away. Or maybe if the professor was kind s/he would mention in passing that I don’t know what the political leanings of the nine porn subscribers are because I haven’t directly analysed them. I fell for the Ecological Fallacy. I incorrectly inferred that the subscribers are defined by the area in which they live.
An important distinction that researchers need to be mindful of.