I continue to get email from supporters of Charles Colson, who keep explaining to me (clearly I’m a slow learner) that it’s perfectly OK to define the “graduation” requirements of a program to include lots of accomplishments that strongly correlate with not committing any more crimes, and then use the fact that “graduates,” so defined, don’t commit many crimes as evidence that the program was a success.
If so, then I have great news to report. I have developed a low-cost program that is 100% guaranteed to reduce recidivism.
Here’s the program: We’ll take a group of people getting out of prison and instruct them (1) to take a Vitamin E capsule every day and (2) not to commit any crimes. We’ll call those who follow the instructions for two years (they tell us they’re taking their vitamins, and that they haven’t committed any crimes, and they haven’t been arrested) “graduates,” and the others (the group consisting of anyone who says he’s stopped taking his vitamins, or admits committing a crime, or has been arrested) “dropouts.” Then we’ll follow the graduates, and the control group, for another year, and see which group gets arrested and re-imprisoned more often.
Bet you anything you like that our graduates do better than random, better even than the graduates of IFI. Because Vitamin E and a warning to stay out of trouble constitutes an effective program? No. Because not getting arrested for two years is a very good predictor of not getting arrested in the third year, just as holding a job and belonging to a church are very good predictors of not getting arrested.
How would we know that the Vitamin E cure was “working” merely through selection effects, rather than having some real impact? Because we would find that the dropout group had a much higher re-offense rate than the controls, just like the dropout group in the IFI study. That’s the telltale sign of “cherry-picking.”
And that’s why an honest evaluation has to follow the drop-outs as well as the graduates. It’s not that we expect the drop-outs to improve; but when we find that the dropout group actually does worse than the control group, then we have to ask whether some or all of the reported difference between graduates and controls reflected selection effects rather than any actual impact of the program.
Previous post, with links, here. [*]