Great seminar yesterday by my colleague Tom Kane. The talk focused on a major problem with the “No child left behind” bill, which proposes to create incentives for schools to perform well by rewarding and punishing schools and their staffs for year-to-year changes in student test scores.
The problem is measurement error, both random and non-random. [This is separate from the problem of systematic cheating due to Dukenfield’s Law of Incentive Management.] The measurement error is large compared to the actual variability among schools; as a result, incentive programs that reward, e.g., “most improved” schools wind up, as Tom said, “Mostly paying for the noise rather than the signal.”
In addition, since the sampling error goes down as sample size rises, small schools are much more likely to be rewarded and much more likely to be punished than larger schools; the principal of a big school has so little chance of getting a prize as to largely eliminate any incentive effect. When schools a rewarded only if every identified ethnic group does well, and punished if any of them does badly, schools with heterogeneous populations will predictably come up on the short end.
Apparently no one in the White House or on the Hill noticed any of this until Tom and his collaborator Doug Staiger pointed it out: after the bill had passed both houses and had gone to conference.