<< Ebola, Race and Criminal Justice | Main | No New SCOTUS Cases >>

Studies and Coding

In almost every debate about policy, someone asserts confidently that "studies show" whatever supports their position.  However, what "studies show" is not necessarily so.  Occasionally on this blog we highlight why the neat little bottom line result that makes the newspaper may not be true.  Sometimes, the reason involves inherent difficulties in studying a particular area, fully disclosed by the researcher, but deemed too complicated for a newspaper article.  Sometimes it is simple sloppiness on the part of the researcher.  In the worst case, it represents intentional manipulation by an agenda-driven researcher intent on producing "evidence" for a predetermined position.

Eugene Volokh has this comment at VC on a study that purports to show that Supreme Court Justices are more likely to vote for protection of a speaker in First Amendment cases if they are ideologically aligned with that speaker.

Before studiers can crunch numbers, they have to reduce real-world realities to simple numbers.  This is called coding and there is a lot of opportunity for either error or distortion in this process.
In his post, Volokh notes a number of questionable codings in this study.  Are the Jehovah's Witnesses really "liberal"?  Is the purveyor of a dogfighting video really "conservative"?

Another problem with liberal/conservative coding is that it assumes a linear, one-dimensional model of politics derived from seating arrangements during the French Revolution.  But are the "extreme right" and "extreme left" really polar opposites, more different from each than either is from the middle?  No.  Hitler and Stalin were two peas in a pod, much more like each other than either was like the western democracies.

The study classifies the "speech" (cross-burning) in Virginia v. Black as "conservative."  Nonsense.  There were two cases joined here.  One was a Klansman burning a cross at a KKK rally.  That ideology is no sense aligned with the beliefs of mainstream American conservatives.  The other case involved two punks who burned a cross to intimidate neighbors, "speech" with no discernible ideological orientation.

So what does a study with such dubious coding actually show?  Nothing, really.

We should be very skeptical of simplistic claims that "studies show" thus and such.  The more political or ideological the subject matter, the more skeptical we should be.


This is why, in death penalty debates for example, abolitionists live off "studies" and would sooner swallow a whale than talk about the facts of specific cases.

I think that at least in modern American parlance there is very little ideological consistency amongst "liberals" and "conservatives". Rather the labels represent coalition of groups in marriages of convenience stuck in a two party system somewhat uncommon in the rest of the Democratic world.

Leave a comment

Monthly Archives