<< Cookie Monster Busted | Main | News Scan >>


Law School Rankings, Data Availability, and What Studies "Show"

| 1 Comment
Last month, US News & World Report released its annual rankings of law schools.  There was considerable turmoil in the middle ranks, as many schools went up or down a substantial number of rungs on the ladder.  See, e.g., this post by Jacob Gershman at WSJ Law Blog.  There is a lesson here on the limitations of studies.
The turmoil is due in considerable part to improved data now available on how many of a law school's recent graduates are actually working as lawyers.  The take-home lesson for the purpose of this blog is that a study can "show" something that is not true because of the limitations of its data.  Hofstra Law did not actually fall 24 places because of a sudden deterioration in its program; it's rating fell 24 places because this year's survey had data that last year's survey lacked.

Every study that attempts to reduce complex human matters to a mathematical equation is an approximation of reality, not reality itself.  "All models are wrong, but some are useful," an old saying goes.  Models are inherently limited, but they are also limited by the available data.

Imagine yourself in an art gallery in complete darkness.  You turn on a red light and look at a painting.  You turn off the red light, turn on a blue light, and look at it again.  You see something entirely different, depending on which photons you have available.  Neither is the real picture.

In public policy debates, including those in criminal law, advocates emphatically proclaim what "studies show" as if they were Revealed Truth.  These claims need to be considered skeptically, both because models are inherently limited and because they are often fed with flawed or incomplete data.  Does a study "show" race affects sentencing by comparing "similar" cases?  That determination of similarity is made with data produced by a coding process that cannot be complete and may be so flawed as to produce a severely distorted picture, like looking at the Mona Lisa in red light.  Or like rating law schools without really knowing how many of their graduates are actually working as lawyers.

1 Comment

Kent,

You're too charitable. What the word "study" actually means when reported, for example, by the NYT or MSNBC, is "a PR release by a pro-defense advocacy group."

How many times have we seen some re-tread blast from DPIC or the ACLU trotted out as a "recently released study by experts?"

It's all a bunch of pro-defendant types congratulating one another on how smart they all are. Yikes.

I must say that one reason I like teaching at a pretty well regarded law school is that I enjoy being the unfortunate object in the punch bowl.

Leave a comment

Monthly Archives