“Garbage in, garbage out”

A few years ago, a young student complained about the readings in our introductory course in American social policy. She wanted to know the right answer, and our syllabus just confused things by including conflicting readings that argued against each other. The often-terrific New York Times reporter Kevin Sack unfortunately channeled her spirit in Wednesday’s paper.

A few years ago, a young student complained about the readings in our introduction to American social policy. She wanted to know the right answer, and our syllabus just confused things by including conflicting readings. The often-terrific New York Times reporter Kevin Sack unfortunately channeled her spirit in Wednesday’s paper.

The story “On Health Plans, the Numbers Fly” starts out fair enough, noting the cacophony of economic analyses of the competing health insurance plans. Sack notes:

Consultants paid by Mr. McCain concluded that his plan would cover 27.5 million of the uninsured. But four health economists who looked into the McCain plan at the urging of David Cutler, a health care advisor to Mr. Obama, reached a far different conclusion. They estimated in a peer-review article in the journal Health Affairs that the number of uninsured would grow by 5 million after five years.

Sack goes on at some length to present the scoop that economists’ political preferences and ideology are correlated with their predictions of what will happen under the McCain and Obama proposed health plans. Such differences always matter. They matter in 2008, too.

He fails to explain where these analytic disagreements arise, where self-serving partisan assumptions might enter, what we should really learn from the competing analyses of the McCain and Obama plans. Sack never engages these studies, beyond reporting the different bottom-line conclusion.

One might argue that such reports are just fodder for campaign commercials, and otherwise unworthy of notice. Thus we have the following:

“It’s garbage in, garbage out,” said Uwe E. Reinhardt, a health economist at Princeton. “Every econometric study is an effort in persuasion. I have to persuade the other guy that my assumptions are responsible. Depending on what I feed into the model, I get totally different answers.”

This is a strikingly undignified statement from a major economist. Sure, econometrics can be cover for political or financial hackery. I’m embarrassed to ponder what antitrust jurors must think after watching two hired-gun experts fighting in a civil case. Health economics is a step above that. At least it should be.

And yeah, it’s hard to reliably forecast what medical spending and uninsurance rates and medical spending will be in 2018. There are major points of uncertainty and disagreement. Competing studies make markedly different assumptions regarding the health insurance market. There are hard questions: Will employers drop coverage if workers will be taxed on it? How will consumers behave within high-deductible plans? Yet we actually know something about consumers’ and firms’ sensitivity to price in purchasing health coverage. We also know something about how health coverage alters consumers’ medical spending and use of preventive care. Some models are more reasonable and informative than others. Some guesses are more educated than others. The best modelers showcase their uncertainties. They discuss how and when these uncertainties matter. The line “nobody knows anything/everyone will say anything” diminishes what (good) policy analysts do, and it diminishes readers’ understanding.

Sack runs over key differences in the kind of insurance consumers purchase, and which consumers purchase it. The McCain plan, independent of its impact on the number of uninsured, is most advantageous to young and healthy individuals, whom it encourages to buy cheaper and barer plans. In addition to insuring more people, the Obama plan would provide greater coverage and protection to people with cancer, diabetes, or other preexisting conditions. These protections operate explicitly through community rating and other requirements. They are also implicit, since individuals with chronic conditions are typically insured within the relatively protected environment of their own or their spouse’s employer-based plans.

The details aren’t easy to explain, but Ann Wilde Matthews’ story in today’s Wall Street Journal shows that it can be done.

New York Times readers already know that there is hype on all sides during a hard-fought election year. Each plan has its expert proponents, many of whom have good things to say, some of whom are crudely biased. It’s the reporter’s job to cut through these biases to explain what’s really at stake.

It’s all too easy to summarily dismiss the good studies with the bad, those scrutinized by peer-review with partisan campaign litter, those which make sense in light of other studies and those that don’t. If everybody’s guilty, nobody is guilty. Lazy readers can therefore stop paying attention. Talk about “garbage in, garbage out…”

Author: Harold Pollack

Harold Pollack is Helen Ross Professor of Social Service Administration at the University of Chicago. He has served on three expert committees of the National Academies of Science. His recent research appears in such journals as Addiction, Journal of the American Medical Association, and American Journal of Public Health. He writes regularly on HIV prevention, crime and drug policy, health reform, and disability policy for American Prospect, tnr.com, and other news outlets. His essay, "Lessons from an Emergency Room Nightmare" was selected for the collection The Best American Medical Writing, 2009. He recently participated, with zero critical acclaim, in the University of Chicago's annual Latke-Hamentaschen debate.