A reader who does medical research in a university setting reports that the NIH/NSF process for choosing research proposals to fund is is fouled-up as the pre-Shinseki military personnel rating system:
“Excellent” is a horrible grade. “Very Good” is abominable.
“Outstanding” is the only passing grade. Everything else is an “F”. The best way to get “Outstanding” is to already have a grant and be a member of the club. “Strongly recommended” means nothing.
If you don’t get into the club…there is always that job driving the mower at the golf course.
“Ambitious” means that you actually want to really test a hypothesis; this is the kiss of death, for one or two of the experiments might not work as anticipated (or you might not get them all done). Silly me. I thought that was the reason for doing the experiments.
My favorite criticism is, “the PI has not shown that he can produce this particular mutant protein,” even if he has already done this dozens of times in related systems. And then there are the reviewers who seem to have read someone else’s proposal.
Moreover, most reviewers cannot or will not say what they mean. Senior faculty advisors are little help; ask five for their take on a criticism and you get five different, often mutually exclusive, answers.
A good evaluation system has headroom: there should always be a score higher than any actual proposal usually receives. Once the top score becomes common, there’s no way for a reviewer to signal real enthusiasm, and the scoring process becomes dominated by whichever judge is most hostile.