Program Evaluation Using Randomized Experimental Designs

What “treatments” change behavior?   Will higher taxes lead golf’s Phil Mickelson to play fewer rounds?  He said so but since we will never observe his 2013 golf schedule had the Bush Tax Cuts lived on, we don’t know.    In the case of energy efficiency, which investments are cost-effective?  In this Opinion Piece, Catherine Wolfram makes the case for implementing more field experiments so that there is a kosher treatment group whose behavior can be compared to a control group.   Macro policy makers can’t run credible field experiments since there is no contemporaneous control group.   If  Congress could randomly raise taxes for half of the population, the CBO could measure whether this treatment group works less. In this case,   we would learn something about how the rest of the population would be affected by increasing taxes on them.   When you can’t randomize incentives, how do we establish cause and effect?  Many economists continue to grapple with this tough issue.

Author: Matthew E. Kahn

Professor of Economics at UCLA.

20 thoughts on “Program Evaluation Using Randomized Experimental Designs”

  1. You could give the actual tax break group real money and give the placebo tax break group Monopoly money!

  2. This might be an entirely rational action on Mickelson’s part. His earnings are not strongly related to the number of tournaments he enters, and only weakly related to the number of top-10 finishes he has. What is predictive of his earnings are his finishes in major tournaments. So, enter the majors and play the others he wants to play for whatever reason. Maybe he likes the course, maybe he wants to get in shape for the next major, maybe he owes the sponsor a favor or three.

    I doubt it would be a rational action on Donnie Duffer’s part, though. Donnie’s earnings are going to be more strongly related to the number of tournament cuts made, which in turn is capped at the number of tournaments entered.

    But forgive me if I don’t really believe that Mickelson is considering a 2% increase in his highest bracket rate is really motivating him to play less tournament golf.

  3. First, without knowing a damn thing about golf or golfers, I am willing to stipulate that Phil Mickelson (is he the new Tiger Woods or something?) would play less golf if we raised his marginal income tax rate. Further, I believe that Rush Limbaugh, Bill O’Reilly, Ann Coulter and their ilk would drastically reduce their output. And I say: GOOD!!

    More seriously: is history irrelevant, all of a sudden? Is it really necessary to run “field experiments” after we ran the back-to-back experiments known as the Clinton and Bush presidencies? You might argue that those experiments involved too many variables to give precise answers to specific questions like “Will the economy collapse if we raise the top marginal rate to 39.6% today?” If that’s what you argue, then I don’t get what clarity a side-by-side experiment would bring you.

    More seriously still: suppose you got to run a field experiment and got the definitive answer that people with incomes over $X will “work less” if you raise the top marginal rate by Y%. So what? I mean, why should I be in favor of those people “working more”, anyway?


  4. “If you add up all the federal and you look at the disability and the unemployment and the Social Security and state, my tax rate is 62, 63 percent,” Mickelson said. “So I’ve got to make some decisions on what to do.”

    Mickelson is worried about his FICA bill??? And his unemployment tax, which is under $500/yr.??? I make his tax rate under 50%, bearing in mind that state income tax is federally deductible.

  5. Look for paired states: VA versus MD, NH versus MA. When they have dramatically different tax/reg regimes, you can make some guesses. Very hard to do it for Cali – nothing compares! Cali can get away with dreadful choices and people will struggle through there. But – Illinois and Indiana have a lot of commonality, and have made some different choices.

    1. Not true that California is unpairable. Las Vegas is right across the border. Lots of wealthy people choose to live there, where the state income tax rate is 0.0%.

      Let me know when you see Phil moving away from his home to save his state income tax (which is vastly greater than the marginal rise in his Federal). He hasn’t done it in all these years up to now.

      Anyway, this whole brouhaha is bullshit. Sam Snead and Ben Hogan didn’t play less golf just because the tax rates back in the Eisenhower/Kennedy years were hugely higher. Nor did Alfred Sloan and Fred Donner (Chairmen of GM) or Thomas J Watson (Chairman of IBM) work less because of those astronomical tax brackets. Nor did Elvis Presley, Smokey Robinson, or The Supremes.

      Just bullshit.

      1. Hmph. I was just in Vegas over the holidays. If anyone wants to live there because of lower taxes, good riddance!! I guarantee, no one here will miss them.

  6. We keep hearing about how the rich are about to go Galt – just like they did throughout the 20th century from FDR through Clinton – but when are they actually going to do it? Lord knows I’m tired of listening to this crap, and would happily bid the Mickelson’s of the world adieu if only we could return to the hideous economic growth of the post-FDR 20th century.

    How can a college professor offer a specific individual as a case study of taxation without bothering to quantify the effect of taxation on that individual? If tax rates matter, and if Mickelson is an example of why they matter, shouldn’t we try to figure out how much money is at stake for poor ol’ Phil and his ilk?

    And Lordy Lord, are people – even University of Chicago people – still talking about the Laffer curve as something with a potential application to current tax rates?

    I could go on regarding the shoddy reasoning and inept economics on display here, but why bother? There’s very little interest in holding Professor Kahn to any reality-based standard – and certainly no interest at all on Professor Kahn’s part.

  7. There was always something I didn’t like about Mickelson….
    Let’s just say he set off my prick-meter for no overt reason.
    There was just something off-putting about him…

    Now I know my prick-meter has a real knack for separating out the dog dirt from the sand…

    But here is thing:
    If Mickelson (45 million a year) plays less that will mean more money for someone else.
    In effect, another person will be over-payed to hit a little ball around with a flat metal stick and call it “hard work”.

    With Phil gone the lucre gets spread around a little more and probably spent more rather than hoarded.
    This cycling of money helps power the recovery.
    That is exactly what we want.

    As for the world being made the less because Phil won’t be swinging his metal sticks as much…
    Oh please. If you think that get a life…
    The world is filled with talented people who will back fill any hole that appears as fast it appears.
    There is nobody so special on this planet that the world won’t continue apace without them…
    Never has been.
    Never will be.

  8. How about: If your preferred method for extracting answers from the real world does not work, you’re using the wrong method.

  9. “If Congress could randomly raise taxes for half of the population, the CBO could measure whether this treatment group works less.”
    The “treatment group” would be furious at having been arbitrarily singled out for a harmful experiment without their consent and in breach of their fundamental right to equality under the law, and would sue the hell out of the government. Tuskegee and the Stanford prison experiment weren’t such great ideas either.
    It appears that “many economists continue to grapple” with the basic ethical constraints of their profession.

  10. Michael — I believe in fact there was a large experiment in income tax differences in 1968 in New Jersey.

    1. I don’t quite get this. The Stanford prison experiment for instance was quite sound epistemologically – but a moral disaster. However, I agree my difficulty with the tax rate experiment is partly epistemic. Outside Stalin’s Soviet Union it’s impossible to keep the groups ignorant of each other, so the experiment is affected by looking over the fence, and the resulting outrage, not just the incentives.

  11. This is simply social science research. Psychology (my field) and probably sociology (not my field so I’m not sure) have much to say about how to conduct it ethically and how to understand the results. Unfortunately it’s expensive and you know how that would play with the no-nothings in Congress. It also doesn’t play well with physicists and biologists who have decided that they have received Science, and all the rest of us are into pseudoscience.

  12. Given the racial and gender disparities in income in the US, you could try using those to see whether people work less if they have a lower incentive to work…

    1. In other words, that’s an example of exactly how not to use passage of a law to do a comparison.

    2. Wrong. That is NOT an example. Other variables were not controlled. E.g.,

      (a) EPA regulations and/or enforcement?
      (b) USDA regulations and/or enforcement?
      (c) California state agricultural regulations and/or enforcement?
      (d) Grocery store consumer protections regulations and/or enforcement?

      And most important, of course, is that the result of the study was NOT that banning of plastic bags was the direct cause. Rather, they found, people didn’t properly wash their reusable bags. Ignorance of proper sanitation is certainly a problem among consumers, irrespective of how they carry their food home from the store.

Comments are closed.