A massive new study reviews the evidence on whether campaigning works. The answer’s bleak.

All the outreach activity by political campaigns, including door to door canvassing, phone banking, direct mail, and even advertising, has basically no effect on voters’ choice of candidate in general elections, according to a striking new academic study.

The new analysis covers 49 field experiments conducted in real US election campaigns, typically run with cooperation from the campaigns themselves.

Campaigns spend millions of dollars during general elections on canvassing; phone banking; advertising on TV, radio, and the internet; and other efforts designed to win over undecided voters and supporters of the opposing candidate. The new study’s authors, UC Berkeley political scientist Joshua Kalla and Stanford professor David Broockman, conclude that essentially no one targeted is persuaded.

This doesn’t mean that political campaigns never matter. Kalla and Broockman find that these activities can persuade voters in primary elections and during ballot-initiative campaigns. Campaigns can still effectively turn out voters whose minds are already made up about a candidate, and voters can and do change their opinions when prompted by politicians they already support (something a previous study of Broockman’s confirmed).

Moreover, it’s possible that even in general elections candidates can persuade voters by varying their positions on issues (say, defecting from their party consensus to a more popular view) or by receiving different media coverage — factors that Kalla and Broockman’s study doesn’t cover.

But the study should force campaigns to seriously reexamine how they’re spending money, and how they try to persuade winnable voters. Running a successful campaign isn’t impossible. It’s just hard, and might require making some unconventional choices.

“Campaigns probably need to get more creative and think more outside the box,” Broockman says. “Whatever box they are working within now doesn’t usually produce results.”

It’s an especially shocking result given the authors’ previous work. Kalla and Broockman conducted a large-scale canvassing experiment, published in 2016, that found that pro-trans-rights canvassers could change Miami residents’ minds about transgender issues by having intense, substantive, 10-minute conversations with them. The persuasive effects of this canvassing were durable, lasting at least three months. The study was, understandably, a big deal: There was a whole This American Life episode made about it. The idea that having deep conversations with people can change their minds seemed incredibly promising.

But now, Kalla and Broockman are finding that this kind of persuasion doesn’t appear to happen during campaigns, at least not very often.

How the study works

The paper is unusual in that it’s not a straightforward review of existing evidence on campaign persuasion. Kalla and Broockman did find 40 existing randomized field experiments wherein campaigns, working in cooperation with academic researchers, randomly subjected a subset of voters to canvasses, phone calls, direct mail, TV or online ads, and other interventions, and compared how the treated and untreated groups responded to later surveys on how they expect to vote in the upcoming election, or how they voted if the election happened already.

“There has been a lot of buzz but not much public information on the experiments political campaigns have been conducting to learn how to persuade voters,” Broockman writes in an email. “The study contains data from inside dozens of those efforts.”

Kalla and Broockman found that, if the campaign action (canvass, phone call, etc.) happens within two months of election day, the average effect on voter preferences was effectively zero. About one in 800 people reached were persuaded, they estimate.

By contrast, when the campaign action happens well before election day, and the effects are measured quickly thereafter, there’s a real impact on opinions — but it disappears before election day. The sooner you get to the election, the more voters get set in their ways and choose candidates by their partisan alignment, and aren’t persuadable by additional campaigning.

Partisanship seems to be the major factor here. While campaigns can rarely persuade people in general elections, Kalla and Broockman do find that campaigning significantly increases support for primary candidates and ballot measures. And while your partisan identity tells you how to vote in general elections, it’s of much less help for ballot initiatives and primary elections. (By definition, the rivals in a primary election come from the same party — so cues from party leadership tend to be subtler.)

“Although campaigns may have some scope for persuasion in competitive primary elections, where there is no partisan cue, in general elections there are few considerations they can provide today’s voters that would lead them to abandon their party,” Kalla and Broockman write.

The authors conducted nine field experiments during the 2015 and 2016 elections

But Kalla and Broockman weren’t content with only 40 existing studies. The evidence they had on hand was useful but very imprecise: For instance, they found that the eight studies measuring the effect of personal canvassing within two months of election day had an average effect of negative 1.9 percentage points, barely larger than the standard error of 1.7 percent. If you use standard techniques to construct a confidence interval, that finding suggests that late-stage canvassing could do anything from hurt candidates by 5.3 points to help them by 1.5 points.

Not satisfied with that kind of uncertainty, Kalla and Broockman teamed up with Working America, the political organizing branch of the AFL-CIO, to conduct nine more field experiments in a 2015 primary, a special election that year, and the 2016 general election. These were large-scale experiments that radically increased the evidence base they had to work with — “by about a factor of 10,” Kalla and Broockman write.

Working America appears to be an unusually effective canvassing group. In the 2015 Democratic primary they studied — for mayor of Philadelphia— Kalla and Broockman find that a Working America canvass six weeks out from the election increased support for their candidate, Jim Kenney (who won the primary and is now mayor), by 11 percentage points a week later; the effects mostly persisted to the end of the campaign.

Similarly impressive numbers came in a 2015 special election for the Washington state legislature, and a turnout experiment in the 2016 general election in North Carolina. These results confirmed that boosting turnout, primary election persuasion, and perhaps persuasion in special elections are possible, especially with a skilled organization.

But evidence of persuasion in general elections remained negligible. Reconducting their meta-analysis with all this new data, Kalla and Broockman concluded, once again, that contacting voters within two months of a general election does not persuade them. The average effect was indistinguishable from zero.

When persuasion could work

So what does this mean for campaigns? One takeaway is that campaigns and non-campaign groups like Working America could do well to focus more of their energy on boosting turnout at the end of a race than persuading voters earlier on. Another is that campaign funders should consider directing more money to primary election and ballot initiatives, where persuasion does appear possible.

But the authors found a couple possible exceptions in general elections too, when persuasion is possible. They were all in somewhat unique circumstances, but nonetheless suggest possible caveats to the study’s overall pessimism.

The first case occurred in the 2008 US Senate race in Oregon. It was an unusual race; Oregon is a reliably blue state that’s had a Democratic governor for the past 30 years, but the two-term incumbent, Gordon Smith, was a moderate Republican. Smith was unusually pro-LGBTQ for a Republican at that time, most notably pushing for sexual orientation to be added to the federal hate-crimes statute. That led to a common misconception that he also favored abortion rights, when he actually identified as pro-life.

With that in mind, Planned Parenthood Advocates of Oregon and NARAL Pro-Choice Oregon worked together on a campaign in which they identified pro-choice voters and targeted them with mailings and phone calls hoping to educate them on Smith’s actual views, and the pro-choice views of his opponent Jeff Merkley. Crucially, the groups worked with political scientists to randomize which voters got these treatments.

Harvard psychologist Todd Rogers and Temple University political scientist David Nickerson evaluated the campaign, and found that it was very effective. Merkley’s vote share rose by 3.4 to 4.5 percentage points among people getting the messages, for a swing in the two-party vote share of up to 9 points. That’s huge, in campaign terms, especially in a close race where the margin of victory was 3 points.

But this was a very strange situation. Senators aren’t normally as out of step with their home states as Smith was with Oregon. There are a few cases where a similar effort might work — say, if Republicans wanted to remind vehemently pro-life North Dakota voters that Democratic Sen. Heidi Heitkamp is pro-choice — but they’re not the norm.

The more intriguing case of persuasion working occurred in North Carolina. Working America, in the effort being studied by Kalla and Broockman, did an experiment early on in the election where canvassers left flyers attacking the Republican governor, Pat McCrory, for his signing of a “bathroom bill” restricting trans rights, which cost the state billions in business. The experiment concluded that the flyer was persuasive for black voters, but not white voters. Working America then adjusted its technique and only handed out the flyer to black households. By the end of the campaign, the effort as a whole was persuasive, due to this targeting.

This is an expensive technique, requiring as it does an additional experiment to identify demographics for whom certain messages work and demographics for whom they don’t, and it relies on North Carolina having voter files that list voters’ races, and on McCrory’s endorsement of a very unpopular law. It’s not easily replicable. But it suggests that building on early studies can help campaigns improve their persuasiveness.

But all that aside, it was a case where persuasion worked. And that matters. Even given its overall low effectiveness, persuasion is an important part of campaigning. Other methods of attempting to gain more votes than your opponent have their own drawbacks. Voter registration is costly; an experiment by Temple’s Nickerson found that voter registration efforts cost about $60 per vote for campaigns running them, far more than turnout efforts. Turnout efforts directed at party bases aren’t enough for candidates running in unfavorable environments, like a Democrat in Alabama or a Republican in Vermont, who need to persuade voters of the other party (and sometimes do: Vermont governor Phil Scott is a Republican). Meanwhile, persuasion efforts can effectively net campaigns two votes, by subtracting one from the opposing campaign too.

All of which makes getting persuasion right important for campaigns. The trouble is that getting it right appears to be really, really hard.