A Republican bubble? How pollsters and pundits got the US midterms so wrong

During the month leading up to the US midterm elections, talk of a commanding Republican victory went from a "red wave" to a "red tsunami." The Republicans were on for the win. The polls and gambling markets, or so-called "prediction markets," were confident.

Only the red wave never broke—Democrats tightened their shaky grip on the Senate and, while they lost control of the House, they did so by a much narrower margin than had been expected.

As part of my research on political betting and gambling markets, I've identified a surge of interest in political gambling since the Brexit referendum and the 2016 US presidential election. Underdog victories in these contests alerted many people in the UK and US—but also internationally—to the opportunity to win big by gambling on politics. And, along with the latest polls, what the betting markets are saying is increasingly considered a good predictor of future events. But not this time.

In the run-up to the US midterm, I was betting (with my own money) against the Republican wave and for a close election—not out of any particular insight so much as caution.

When the results began to trickle in and it became clear the predicted Republican takeover was not happening, I had an unexpectedly successful few days of profits. Meanwhile, I frantically tried to figure out why the betting markets had predicted otherwise and what this failure meant.

Why so wrong?

To understand why the gambling markets got it so wrong, we first need to look at what evidence there was for a red wave.

1. History says so

First up there's historical precedence. The party of a first-term US president almost always loses significant numbers of seats in both houses of Congress in the midterms two years after they are elected.

2. The polls tightened

The polls also indicated that a red wave could happen. Democrats took the lead in the polls in mid-June, but the lead started narrowing in mid-September, with the Democrats and Republicans tied on 50 senate seats each on November 1.

3. Predictions went red

While some polls indicated a tight race, organizations using more complex predictive models swung towards Republicans. By election day, FiveThirtyEight, the highest-profile of these organizations, was predicting the Republicans would take control of the Senate 59 times out of a hundred—and people listened.

4. The odds were high

This meant the markets were heavily favoring Republicans by late October. On the UK site Betfair, the world's largest betting exchange, the likelihood of a republican majority shot above 50% on October 19 and peaked at 78% on election day—only to crash to 12% a day later as results began to become clear. On the foremost US provider, PredictIt, Republicans were trading at around 75 cents a share (a winning share returns US$1, a losing share 0 cents) before they, too, crashed in the face of election count data.

Inflated victory

Now looking back, it's clear that a market bubble had inflated around a Republican victory. One of the maxims repeated in political betting circles is "bet the trend, not the poll" and the trend, as evidenced in the polling, started shifting sharply towards Republicans before then leveling off. Betters and modelers projected the original trend towards Republicans and ignored the leveling off.

There were some in the community who were arguing against the crowd—that the odds had shifted too far towards the Republicans. But their voices were drowned out in a sea of optimism (or pessimism, depending on your politics). Indeed, Matthew Shaddick, head of politics at the UK betting exchange Smarkets, spoke about it on the company podcast. He said that the last month before the election was "one-way traffic" with everyone wanting to back Republicans.

The sophistication and budgets of election campaigns in the US are also so extensive that considerable effort is made to influence the narrative through polling. According to my contacts, there was a flurry of Republican-leaning polls that were pushing their chances. And, as we know, polls influence people's decisions when it comes to betting.

I'm also often asked whether political parties might bet on themselves to improve the perception of their campaign. While this is less likely to have any consequence with larger events such as the US elections, as with polling, at a smaller scale it can have an effect on a candidate's implied probability of winning, which can then filter into the media.

There is more research to be done, but this failure of prediction could not come at a worse time for US political gambling providers, styled as prediction markets. They have been trying to convince a skeptical regulator, the Commodity Futures Trading Commission, that political gambling markets add value to politics and financial traders because of their predictive potency. Indeed, gambling markets are usually considered much more accurate than polls, but it's hard to see the midterms as anything other than a failure of prediction.

This article is republished from The Conversation under a Creative Commons license. Read the original article.