Incentivizing randomness

Great seminar yesterday by my colleague Tom Kane. The talk focused on a major problem with the “No child left behind” bill, which proposes to create incentives for schools to perform well by rewarding and punishing schools and their staffs for year-to-year changes in student test scores.

The problem is measurement error, both random and non-random. [This is separate from the problem of systematic cheating due to Dukenfield’s Law of Incentive Management.] The measurement error is large compared to the actual variability among schools; as a result, incentive programs that reward, e.g., “most improved” schools wind up, as Tom said, “Mostly paying for the noise rather than the signal.”

In addition, since the sampling error goes down as sample size rises, small schools are much more likely to be rewarded and much more likely to be punished than larger schools; the principal of a big school has so little chance of getting a prize as to largely eliminate any incentive effect. When schools a rewarded only if every identified ethnic group does well, and punished if any of them does badly, schools with heterogeneous populations will predictably come up on the short end.

Apparently no one in the White House or on the Hill noticed any of this until Tom and his collaborator Doug Staiger pointed it out: after the bill had passed both houses and had gone to conference.

Author: Mark Kleiman

Professor of Public Policy at the NYU Marron Institute for Urban Management and editor of the Journal of Drug Policy Analysis. Teaches about the methods of policy analysis about drug abuse control and crime control policy, working out the implications of two principles: that swift and certain sanctions don't have to be severe to be effective, and that well-designed threats usually don't have to be carried out. Books: Drugs and Drug Policy: What Everyone Needs to Know (with Jonathan Caulkins and Angela Hawken) When Brute Force Fails: How to Have Less Crime and Less Punishment (Princeton, 2009; named one of the "books of the year" by The Economist Against Excess: Drug Policy for Results (Basic, 1993) Marijuana: Costs of Abuse, Costs of Control (Greenwood, 1989) UCLA Homepage Curriculum Vitae Contact: