Peer review under high pressure

… doesn’t work so well, says a working scientist.

Commenting on my remark that the peer review system for scientific funding tries to pick out the projects with the greatest scientific merit, a reader who is a working bioscientist writes [note: Elias Zerhouni is the current Director of the National Institutes of Health] :

Not really, Mark. Not now. I don’t know how much public policy research depends on extramural funding, but the “peer review system” does not work as you (or Michael O’Hare) say. It can’t. Remember your comment last year about a scientist at UCLA not getting funded even though the grant was judged to be better than 94% of all proposals in that particular study section? It has gotten worse since then, at least in the study sections I use, whatever Zerhouni and whichever bureaucrat is currently in charge at NSF say.

When paylines are so low, whoever gets funded is either stochastic or predetermined, as some of my colleagues here and at other institutions in fly-over country believe: especially for those scientists at the large private universities on the coasts; some might include UCLA, Berkeley, and UCSF in that group. Me? Well, I wouldn’t kill just anybody for a job at UCSF.

Relative merit as determined by so-called peer review has no real meaning if paylines fall much below 33%. Period. Subjective considerations, primarily involving what a particular review panel deems to be “sexiest” at a given moment are the determining factors as the pool of money dwindles. Therefore, for those scientists who score an earmark, all I can say is “More power to you!” That, and use the money wisely (yeah, I know: unlikely). Any port in a storm, and this is a real storm.

Ask your friends in the UCLA medical school what they think of Zerhouni’s so-called revamping of peer review…or his Roadmap…or the emphasis on “translational research.” On the latter I recommend a recent article in Science on the length of time from discovery to translation and beyond (September 5, p 1298).

The good news is that we’re now generating many more good research ideas than current funding will cover. But that’s good news only if we start to expand the budget. NIH has a budget is just under $30B/yr. NSF, which has to cover all of natural and social science, comes in under $7B. In Federal budget terms, that’s chump change. Here’s hoping that Barack Obama won’t let himself be talked into the idea that we “can’t afford” science.

Author: Mark Kleiman

Professor of Public Policy at the NYU Marron Institute for Urban Management and editor of the Journal of Drug Policy Analysis. Teaches about the methods of policy analysis about drug abuse control and crime control policy, working out the implications of two principles: that swift and certain sanctions don't have to be severe to be effective, and that well-designed threats usually don't have to be carried out. Books: Drugs and Drug Policy: What Everyone Needs to Know (with Jonathan Caulkins and Angela Hawken) When Brute Force Fails: How to Have Less Crime and Less Punishment (Princeton, 2009; named one of the "books of the year" by The Economist Against Excess: Drug Policy for Results (Basic, 1993) Marijuana: Costs of Abuse, Costs of Control (Greenwood, 1989) UCLA Homepage Curriculum Vitae Contact: Markarkleiman-at-gmail.com