More grade inflation: research funding reviews

Grade inflation in competition means that the low-scoring judge is the boss.

A reader who does medical research in a university setting reports that the NIH/NSF process for choosing research proposals to fund is is fouled-up as the pre-Shinseki military personnel rating system:

“Excellent” is a horrible grade. “Very Good” is abominable.

“Outstanding” is the only passing grade. Everything else is an “F”. The best way to get “Outstanding” is to already have a grant and be a member of the club. “Strongly recommended” means nothing.

If you don’t get into the club…there is always that job driving the mower at the golf course.

“Ambitious” means that you actually want to really test a hypothesis; this is the kiss of death, for one or two of the experiments might not work as anticipated (or you might not get them all done). Silly me. I thought that was the reason for doing the experiments.

My favorite criticism is, “the PI has not shown that he can produce this particular mutant protein,” even if he has already done this dozens of times in related systems. And then there are the reviewers who seem to have read someone else’s proposal.

Moreover, most reviewers cannot or will not say what they mean. Senior faculty advisors are little help; ask five for their take on a criticism and you get five different, often mutually exclusive, answers.

A good evaluation system has headroom: there should always be a score higher than any actual proposal usually receives. Once the top score becomes common, there’s no way for a reviewer to signal real enthusiasm, and the scoring process becomes dominated by whichever judge is most hostile.

Author: Mark Kleiman

Professor of Public Policy at the NYU Marron Institute for Urban Management and editor of the Journal of Drug Policy Analysis. Teaches about the methods of policy analysis about drug abuse control and crime control policy, working out the implications of two principles: that swift and certain sanctions don't have to be severe to be effective, and that well-designed threats usually don't have to be carried out. Books: Drugs and Drug Policy: What Everyone Needs to Know (with Jonathan Caulkins and Angela Hawken) When Brute Force Fails: How to Have Less Crime and Less Punishment (Princeton, 2009; named one of the "books of the year" by The Economist Against Excess: Drug Policy for Results (Basic, 1993) Marijuana: Costs of Abuse, Costs of Control (Greenwood, 1989) UCLA Homepage Curriculum Vitae Contact: Markarkleiman-at-gmail.com