<< Evil or Crazy? | Main | Justified Use of Force >>

Studies Are Usually Bunk, Study Shows

Andy Kessler has this article with the above title in the WSJ.

Hands down, the two most dangerous words in the English language today are "studies show."

The world is inundated with the manipulation of flighty studies to prove some larger point about mankind in the name of behavioral science. Pop psychologists have churned out mountains of books proving some intuitive point that turns out to be wrong. It's "sciencey," with a whiff of (false) authenticity.
*              *              *
Bunk medical studies are worrisome, but who really cares about pop behavioral science? It's easy to write this off as trivial, except millions take these studies and their conclusions seriously. The 2008 book "Nudge," from academics Richard Thaler and Cass Sunstein, called for "libertarian paternalism" to push people in the right direction. But who decides what's the right direction? Turns out the answer is Mr. Sunstein. He was hired by the Obama administration in 2009 to run the Office of Information and Regulatory Affairs. Call it psychobabble authoritarianism.
I would not go so far as to say that most studies are bunk.  A far more common problem is people citing studies for their bottom-line result with far more certainty than the study actually warrants.  This unfortunate tendency has multiple causes.  Part of it is lack of sophistication among people who consider themselves oh-so-sophisticated in holding forth on what "studies show."  Understanding the limitations requires depth, and pseudosophisticates don't have it.

Part of the problem is confirmation bias.  If a study "shows" what one already believes, those pesky limitations just clutter the message.

Part of the problem is the need of news media to make their stories simple and newsworthy.  A story that "Study Shows X" is simpler for the readers and bigger news than "Study Suggests X But Then Again Maybe Not For The Following Complex Reasons."

Some studies are bunk, of course, or at least borderline bunk.  Particularly for studies with political implications, advocate-researchers may overstate the certainty and downplay the limitations on purpose.  It is hard enough to understand the limitations of studies when they are stated frankly, but it is much more so when the author attempts to cover them.  This is unethical, but if a study has a Politically Correct bottom line, few people in academia want to risk pointing out its shortcomings or even its dishonesty.

And I do like the phrase "psychobabble authoritarianism."  It is a valuable addition to the lexicon.

Leave a comment

Monthly Archives