<< Not to Say I Told You So About Sentencing... | Main | News Scan >>

Peer Review, Corruption, Junk Science, and Advocacy Science

| 1 Comment
Last week, I had a post about the malleability of computer models and the problem of models done by advocates with no effective check in our Politically Correct academia.  Bill had a post on the scandal of peer-review corruption.

Today in the WSJ, Hank Campbell has an op-ed titled The Corruption of Peer Review Is Harming Scientific Credibility, subtitled Dubious studies on the danger of hurricane names may be laughable. But bad science can cause bad policy.  See also James Taranto's Best of the Web column today, also in the WSJ.

Campbell notes, "in biomedicine faulty research and a dubious peer-review process can have life-or-death consequences."  That is also true in criminal justice.  There is now a concerted effort to affect public policy with studies and papers downplaying the public-safety effect of punishment and touting alternatives.  Can we trust any of this?  Can we trust it with our lives?
Campbell notes the scandal of the peer review ring in the Journal of Vibration and Control, and he also notes the earlier, more dangerous scandal in drug research when 2/3 of studies studied could not be reproduced.  (In hard sciences, "cannot be reproduced" means garbage at best and possibly fraud.  Cold fusion, for example, could not be reproduced.)

Campbell goes on to note two problems that have concerned me for some time -- insufficient transparency and data sharing and the too-easy imprimatur of prestigious organizations, especially government-sponsored ones.

The authors pointed to several reasons for flawed studies, including "poor training of researchers in experimental design," an "emphasis on making provocative statements," and publications that don't "report basic elements of experimental design." They also said that "some scientists reputedly use a 'secret sauce' to make their experiments work--and withhold details from publication or describe them only vaguely to retain a competitive edge."

Papers with such problems or omissions would never see the light of day if sound peer-review practices were in place--and their absence at many journals is the root of the problem. Peer review involves an anonymous panel of objective experts critiquing a paper on its merits. Obviously, a panel should not contain anyone who agrees in advance to give the paper favorable attention and help it get published. Yet a variety of journals have allowed or overlooked such practices.

Absent rigorous peer review, we get the paper published in June in the Proceedings of the National Academy of Sciences. Titled "Female hurricanes are deadlier than male hurricanes," it concluded that hurricanes with female names cause more deaths than male-named hurricanes--ostensibly because implicit sexism makes people take the storms with a woman's name less seriously. The work was debunked once its methods were examined, but not before it got attention nationwide.

Such a dubious paper made its way into national media outlets because of the imprimatur of the prestigious National Academy of Sciences.

Yet a look at the organization's own submission guidelines makes clear that if you are a National Academy member today, you can edit a research paper that you wrote yourself and only have to answer a few questions before an editorial board; you can even arrange to be the official reviewer for people you know. The result of such laxity isn't just the publication of a dubious finding like the hurricane gender-bias claim. Some errors can have serious consequences if bad science leads to bad policy.

In 2002 and 2010, papers published in the Proceedings of the National Academy of Sciences claimed that a pesticide called atrazine was causing sex changes in frogs. As a result the Environmental Protection Agency set up special panels to re-examine the product's safety. Both papers had the same editor, David Wake of the University of California, Berkeley, who is a colleague of the papers' lead author, Tyrone Hayes, also of Berkeley.

In keeping with National Academy of Sciences policy, Prof. Hayes preselected Prof. Wake as his editor. Both studies were published without a review of the data used to reach the finding. No one has been able to reproduce the results of either paper, including the EPA, which did expensive, time-consuming reviews of the pesticide brought about by the published claims. As the agency investigated, it couldn't even use those papers about atrazine's alleged effects because the research they were based on didn't meet the criteria for legitimate scientific work. The authors refused to hand over data that led them to their claimed results--which meant no one could run the same computer program and match their results.

A while back, there was a government-funded project called the Capital Jury Project.  The articles that came out of it had a pro-defendant, anti-death-penalty orientation because all or nearly all of the researchers with access to the data had that orientation.  I asked the keeper of the data, Theodore Eisenberg, for access so I could do my own research, and he turned me down flat.  His excuse was that the study participants had been promised confidentiality.  That is a preposterous excuse.  Protecting participant confidentiality is very important, of course, but methods for doing so while still sharing the data by "anonymizing" it are well established.  The real reason, almost certainly, is to keep the data safely in the hands of those who will use only to produce Politically Correct results.

There was an attack on the death penalty deterrence studies published in 2012 under the imprimatur of the National Academy of Sciences.  The acknowledgements paragraph in the preface revealingly thanks several of the authors of the anti-deterrence studies and none of the authors of the pro-deterrence studies.  Were the composition of the committee and the review process manipulated to insure the conclusion remained anti-deterrence?  James Q. Wilson was on the committee, even though this is not his field of expertise, but unfortunately he died before the final report was completed, so we will never know if he approved of it.  The report begins with an effusive dedication to him, and methinks the author doth effuse too much.  Campbell's comments about NAS deepen my existing suspicion about the process that produced this report.

Studies and reports in criminal justice and corrections need some hard, clear-eyed review by people with the expertise to do it and the dedication to the truth to do it without bias.  That appears to be sadly lacking.

1 Comment

These statistics posts are all very fascinating. I think the analysis of what prompts a prosecutor to seek the death penalty is not all that interesting and probably has a very obvious answer - prosecutor's offices have limited resources to pursue to capital cases, therefore it would stand to reason the driving factor would be the presence of many aggravating circumstances and the absence of many mitigating circumstances.

I think the analysis is far more interesting from a "moneyball" perspective where an analysis is done of what constitutes a successful capital prosecution - i.e. what did the prosecutor do to secure the conviction versus what happened in an unsuccessful capital prosecution. I think it'd be a fascinating study for both prosecution and defense.\

Leave a comment

Monthly Archives