Ted Kennedy was an extremely effective Senator, but he correctly identified as a disastrous decision his refusal to work with President Nixon on universal health insurance. At the time, Kennedy assumed that the liberal wave that had started under LBJ (who of course created Medicare and Medicaid with Kennedy’s support) was about to become a post-Watergate tidal wave, making a much more generous health care reform package possible very soon. Imagine Senator Kennedy’s pain seeing liberalism’s subsequent retreat and enduring a more than three decade wait before a similar health policy became possible.
More recently, California prosecutors pressured Governor Jerry Brown into vetoing a 2013 legislature-passed bill that would have reduced simple drug possession from a felony to a “wobbler” (something that could be charged either as a misdemeanor or felony). The public responded the following year by overwhelmingly passing a much more sweeping sentencing reform by ballot initiative, making all simple drug possession chargeable as a misdemeanor only as well as reducing criminal penalties for a range of other crimes.
Why do political actors so often believe that they have a powerful hand when in fact they are grossly out of step with the times and would much better off accepting half a loaf while they can? The simple explanation is arrogance, but politics includes many arrogant people who also have a keen sense of whether the public is with them on some issue. Perhaps insularity is the culprit. Many political actors live in an echo chamber that makes them overestimate the popularity of their views. If being in an information bubble makes you believe that your complete political triumph is just around the corner, why compromise?
However, Andy Sabl, who has forgotten more about political science than I will ever know, pointed out to me a flaw in the information bubble explanation: Sometimes misreading of the future drives rather than prevents political compromise. Both sides for example could be thinking “We can get much of what we want right now except for these few stinking compromise amendments, but we will wipe those out after our big win in the next election”. Of course, both sides can’t be correct that they will win big in the next election (indeed they could both be wrong), but the inaccurate expectations may make them happy to take half a loaf because they soon expect it to double in size.
If the information bubble hypothesis is wrong (or at least incomplete), another plausible mechanism is that sometimes political actors know they should compromise, but scorch the earth instead because of other incentives. For example, if you are the lead lobbyist for the California prosecuting attorneys, you may be more likely to lose your job if you say “I cut a deal because the future is against you” versus “I succeeded in influencing the legislature as I promised, but then the other side pulled an underhanded trick and used a ballot initiative, which is not something for which you can blame me”.