<< News Scan | Main | Attorney General Sessions Speaks at Georgetown >>

The "Studies Show" Shell Game: Minimum Wage Version

David Neumark, professor of economics and director of the Economic Self-Sufficiency Policy Research Institute at the University of California, Irvine, has this op-ed in the WSJ regarding the empirical research on the effects of boosting the minimum wage.  The particular issue is, of course, off-topic for the blog, and CJLF takes no position on it.  The tone of the debate, though, is all too familiar to those of us who study studies in criminal law:

Does the minimum wage destroy jobs? The debate over that question often reduces to dueling economic studies. One side cites analyses showing that employers respond to a wage floor by cutting hours or jobs. The other side pulls out studies saying the minimum wage is a free lunch for workers. To really understand what's going on, you need to get under the hood.

The key challenge in estimating the effects of a rising minimum wage is identifying a good control group. Generally economists want to find a set of workers who weren't subject to the policy change, but who otherwise experienced similar economic trends. Still, that leaves a lot of leeway for choice.

There are two big differences between physical and social science.  You can run an experiment with quadrillions of subatomic particles and carefully control the conditions, so that you have two otherwise truly identical groups with the variable you are measuring as the only difference.  Humans object to being controlled like that, and you can't have that many of them.

The other big difference is that if you are studying the spin of quarks you won't get demonstrators outside your office and half of your colleagues signing letters against you if you come up with the "wrong" answer.  Some of the greatest moments in the history of science have been experiments that showed that what "everybody knows" is not actually true.

Neumark takes us "under the hood" to the extent he can in a general-audience newspaper.  He describes the "close comparison" method, where a study compares neighboring areas separated by a jurisdictional line where one adopts the change in policy and the other does not.  Sound familiar?

Critics say these studies do not convincingly control for shocks to the low-skill labor market. Moreover, comparing across state borders is inherently difficult. Perhaps politicians in one state felt comfortable raising the minimum wage because the labor market there was already strong, while the other state was struggling. In that case, job losses from the higher minimum wage could be masked by the broader trend.
That is exactly the approach we sometimes see in crime studies and exactly the problem with it.  What happens if someone else studies the same problem with a different method?  Often you get a different result.

Alternative research strategies do show job losses. Consider a 2014 study by Jeffrey Clemens and Michael Wither on what happened when the federal minimum wage was raised from $5.15 to $7.25 in the late 2000s. Rather than crossing a border, they compared groups of workers within each of the affected states. The first group included those who were paid the very lowest wage, on whom the new minimum was "binding." The control group included workers earning slightly above the minimum. The study estimated that the new federal minimum had eliminated about 800,000 jobs.
Are criticisms of methodology good-faith technical disagreements, or are they simply weapons for people whose real agenda is to shoot down studies that run contrary to positions ultimately grounded in ideology?  Sometimes one, sometimes the other.

The dispute over methodology explains the importance of this summer's research on Seattle's minimum-wage experiment. The city's wage floor, previously about $9.50 an hour, has been raised to $13 and is on its way to $15. A comprehensive study by academics at the University of Washington estimated that the higher minimum "reduced hours worked in low-wage jobs by around 9 percent." Consequently, earnings for these employees actually dropped "by an average of $125 per month."

What's especially inconvenient for minimum-wage proponents is that the Seattle study used a "close comparison" method similar to the one they have favored for years. The authors of the study compared workers in Seattle with those in other metropolitan areas in Washington, like Olympia, Tacoma and Spokane.

To no one's surprise, that hasn't stopped minimum-wage supporters from attacking the Seattle research. In a June letter to city officials, Mr. Reich, the Berkeley professor, wrote that the study "draws only from areas in Washington State that do not at all resemble Seattle." But this gives away the game: Any researchers doing this kind of study should explicitly choose control areas that show similar trends, as did the University of Washington team. More to the point, if the controls for Seattle can't be trusted, it undermines the whole idea of "close comparison." Criticizing the method only when it delivers evidence against minimum wages suggests the motivations here may be ideological rather than empirical.

The vicious attacks on capital punishment deterrence research may be similar.  I don't pretend to have the technical expertise to evaluate the merits of that debate, but there are indications we should be deeply skeptical of the attacks.  The NRC committee report by Nagin et al. has a smoking gun buried in the acknowledgements, of all places.  They committee gratefully acknowledges the assistance of the persons asked to review and comment on the report.  First on the list is the most shrill, strident partisan of one side of the debate.  Nobody from the other side is listed.

Back to the minimum wage:

The Seattle study will not settle the debate--nor should it. It is one analysis, and it examines one city's experience over a short period. In my view, more of the evidence, and the best evidence, points to job losses from higher minimum wages. But additional research will refine the academic consensus. The trouble is that the debate will suffer if researchers use their influence to attack results they don't like, as seems to be happening with the Seattle study.


There is also a more general lesson for the public. Although politicians and the media are right to rely on research to understand the effects of public policy, it's important to keep a skeptical eye. Research often comes from a mix of sources, and it can be difficult for those reporting on it to separate the wheat from the chaff. Perhaps it's even unreasonable to expect them to do so.

This problem of research-as-advocacy isn't confined to one side. There is plenty of it on the right and the left. Yet it would be wrong, too, to think that any study from an organization with a point of view is flawed or biased. The trick is trying to understand not only what the research says about public policy, but also who the messenger is.

There are a number of organizations with clear soft-on-crime agendas that regularly pump out studies claiming to show results consistent with their agenda.   The press needs to be more aware of where they are coming from and inform the public of their agenda whenever research results are reported.  The counter-intuitive claims that we can release large numbers of the people presently in prison without increasing crime often come from such organizations, but they are typically identified in news reports with neutral designations.  On the other side there is very little.  CJLF is a public-interest law firm, neither staffed nor funded to do sophisticated empirical research.  Among academics, the enormous imbalance in viewpoint, which is even greater among criminologists than in the academy generally, further aggravates the problem.

In criminal law we haven't seen much research-as-advocacy on the tough-on-crime side, although that is not to say it can't happen.  We need to have reviewers dedicated to the truth for its own sake, letting the results fall where they may.  Sadly, we cannot count on the institutions we used to count on.  They are regularly being captured by people who place politics above truth.  Where the government funds research or reviews of research, it needs to guard against ideological forces distorting results.

Leave a comment

Monthly Archives