Elsevier

International Journal of Forecasting

Volume 35, Issue 1, January–March 2019, Pages 336-350
International Journal of Forecasting

Polls to probabilities: Comparing prediction markets and opinion polls

https://doi.org/10.1016/j.ijforecast.2018.04.001Get rights and content

Abstract

The forecasting of election outcomes is a hugely popular activity, and not without reason: the outcomes can have significant economic impacts, for example on stock prices. As such, it is economically important, as well as of academic interest, to determine the forecasting methods that have historically performed best. However, the forecasts are often incompatible, as some are in terms of vote shares while others are probabilistic outcome forecasts. This paper sets out an empirical method for transforming opinion poll vote shares into probabilistic forecasts, and then evaluates the performances of prediction markets and opinion polls. We make comparisons along two dimensions, bias and precision, and find that converted opinion polls perform well in terms of bias, while prediction markets are good for precision.

Introduction

Electoral outcomes can have significant economic impacts, and as such, it is economically important, as well as of academic interest, to evaluate forecasting methods for elections. This article compares the forecasts of US election outcomes from a number of prediction markets and opinion polls.

Opinion polls are surveys of the voting intentions of a sample of voters, while prediction markets allow participants to trade contracts of which the value is contingent on some particular outcome occurring. The largest commercial prediction markets over this period produced predicted probabilities, whereas opinion polls produced projected vote shares. Thus, we begin by developing an empirical method for converting vote shares into outcome probabilities, then compare the two sources of forecasts by considering both the bias and the precision, which are reflected elegantly in the commonly-used Brier score, or mean squared error.

We consider all opinion polls from two common aggregators of polling information (Real Clear Politics and Pollster), and look at three well-known prediction markets: Intrade, Betfair and the Iowa Electronic Markets (IEM, henceforth). There have already been several academic investigations into the performances of opinion polls and prediction markets (e.g. Berg et al., 2008, Kou and Sobel, 2004, Leigh and Wolfers, 2006, Rothschild, 2009), and we contribute to this growing body of evidence.

While prediction markets have a long and rich history (Rhode & Strumpf, 2013), their internet-based electronic variants have been garnering increasing amounts of attention over recent years, with 2012 arguably marking the first year in which one of them, Intrade, was in the news regularly alongside traditional polling information. Fig. 1 provides some background on this, as it reports the relative search frequencies on Google in the US for Gallup (a poling company with a long history), Intrade and Betfair (the two best known prediction markets). Google search information is available for the period since early 2004, and hence, three election cycles, those in 2004, 2008 and 2012, are distinctly visible from the spikes in search volumes, most pertinently for Gallup, but also for Intrade. Gallup registered more than sixty times as many searches as Intrade around the 2004 election, but only ten times as many in 2008, and three times as many in 2012.

The 2012 election, and to a lesser extent the 2004 and 2008 elections, also bore witness to a distinct divergence between the two most commonly known prediction markets, Intrade and Betfair, with Republican presidential candidates tending to be priced more favourably on Intrade than on Betfair. This divergence was noted by Rothschild and Sethi (2016), who analysed the behaviour of one particular trader who lost around $4m, apparently in manipulating the market price in favour of the Republican candidate, Mitt Romney. A clear implication of the Rothschild and Sethi (2016) paper is that there was a distinct bias on Intrade relative to Betfair (and also to the Iowa Electronic Markets); investigating the bias forms part of our description of each market.

Thus, this paper enables an enhanced comparison of opinion polls and prediction markets by providing an empirical method for converting polls into probabilities. We also build on previous studies that have compared opinion polls and prediction markets by considering a range of prediction markets, rather than a single one. Section 2 introduces our datasets, giving some details about the nature of each source of forecasts, and the information we have for each source. Section 3 discusses our methodology for transforming vote-share polls into probabilistic forecasts, and subsequently for appraising each forecast source based on its bias and precision. Section 4 provides the results, comparing our prediction markets to each other and to opinion polls, and Section 5 concludes.

Section snippets

Data

Data are fundamental to this investigation, and hence we begin by introducing our data and sources, before discussing our methods. As our primary aim is to convert opinion polling data into a form that can be compared more readily to prediction market data, we start by introducing our opinion polling data in Section 2.1. We then introduce prediction market data from Betfair (Section 2.2.1), Intrade (Section 2.2.2) and IEM (Section 2.2.3). It is important to be clear at the outset: our selection

Methodology

This section sets out our methodological approach to the comparison of opinion polls and prediction markets. We consider the bias and precision of the two methods, but need to convert opinion polls from vote share projections into probabilistic forecasts. As part of this, we bias-correct opinion polls, and hence, it is appropriate to also bias-correct prediction market forecasts in order to facilitate more insightful comparisons.

Ideally, opinion polls would be random samples of voting

Converting polls to probabilities

The first two columns of Table 1 report the results of the probit regression that converts the polled vote shared into implied probabilities of an election victory for the candidates (Eq. (2)). Each column corresponds to a particular candidate, Republican or Democrat, respectively. The first row of the table is the constant term, while the subsequent three rows show the impact on the likelihood of victory of additional percentage points for Democratic, Republican and Independent candidates in a

Conclusions

Election outcomes matter for economic outcomes, and as such, it is important to determine effective ways of forecasting electoral outcomes. This paper provides an empirical method of transforming opinion poll vote shares into outcome probabilities in order to allow a comparison between prediction markets and polls around the 2008–2012 US election cycle. We consider a richer range of prediction markets than most previous studies, evaluating Intrade, Betfair and Iowa Electronic Markets, as well

James Reade is an Associate Professor in Economics at the University of Reading. Prior to this he was a Lecturer in Economics at the University of Birmingham, and completed his Ph.D. in Economics at the University of Oxford in 2007. His research interests are in sports economics and forecasting.

References (18)

  • BergJ. et al.

    Prediction market accuracy in the long run

    International Journal of Forecasting

    (2008)
  • BrierG.

    Verification of forecasts expressed in terms of probability

    Monthly Weather Review

    (1950)
  • EriksonR. et al.

    Are political markets really superior to polls as election predictors?

    Public Opinion Quarterly

    (2008)
  • GelmanA. et al.

    Why are American presidential election campaign polls so variable when votes are so predictable?

    British Journal of Political Science

    (1993)
  • Graefe, A. (2013). Accuracy of vote expectation surveys in forecasting elections. Working paper, Department of...
  • HayekF.

    The use of knowledge in society

    The American Economic Review

    (1945)
  • HendryD. et al.

    Combining disaggregate forecasts or combining disaggregate information to forecast an aggregate

    Journal of Business & Economic Statistics

    (2011)
  • HurleyW. et al.

    A note on the Hayek hypothesis and the favorite-longshot bias in parimutuel betting

    The American Economic Review

    (1995)
  • KouS. et al.

    Forecasting the vote: A theoretical comparison of election markets and public opinion polls

    Political Analysis

    (2004)
There are more references available in the full text version of this article.

Cited by (7)

  • Crowd prediction systems: Markets, polls, and elite forecasters

    2024, International Journal of Forecasting
  • A stochastic differential equation approach to the analysis of the 2017 and 2019 UK general election polls

    2021, International Journal of Forecasting
    Citation Excerpt :

    However, it is not clear whether prediction markets perform better than polls. In Reade and Vaughan Williams (2019), the authors concluded that opinion polls are favourable in terms of their bias (the mean error of all forecasts), while prediction markets are better in terms of their precision (the reciprocal of the variance of all forecasts). Citizen forecasting is the process of aggregating forecasts made by individuals, which can be viewed as a form of ‘wisdom of crowds’.

  • Towards Conditional Prediction Markets as Policy-Making Instruments for the Promotion of Renewable Energy

    2022, SEST 2022 - 5th International Conference on Smart Energy Systems and Technologies
View all citing articles on Scopus

James Reade is an Associate Professor in Economics at the University of Reading. Prior to this he was a Lecturer in Economics at the University of Birmingham, and completed his Ph.D. in Economics at the University of Oxford in 2007. His research interests are in sports economics and forecasting.

Leighton Vaughan Williams is Professor of Economics and Finance and Director of the Betting Research Unit and of the Political Forecasting Unit at Nottingham Business School, Nottingham Trent University.

View full text