Forecasting proportional representation elections from non-representative expectation surveys

Highlights

• Expectation surveys work well with non-representative samples for forecasting proportional representation elections.

• Expectation surveys are a useful low-cost method for forecasting elections, in particular small-scale elections.

• Surveying more interested and knowledgeable citizens improves forecast accuracy.

• Future research is necessary to assess the predictive value of non-representative expectation surveys.

Abstract

This study tests non-representative expectation surveys as a method for forecasting elections. For dichotomous forecasts of the 2013 German election (e.g., who will be chancellor, which parties will enter parliament), two non-representative citizen samples performed equally well than a benchmark group of experts. For vote-share forecasts, the sample of more knowledgeable and interested citizens performed similar to experts and quantitative models, and outperformed the less informed citizens. Furthermore, both citizen samples outperformed prediction markets but provided less accurate forecasts than representative polls. The results suggest that non-representative surveys can provide a useful low-cost forecasting method, in particular for small-scale elections, where it may not be feasible or cost-effective to use established methods such as representative polls or prediction markets.

Introduction

Response rates in traditional phone surveys have decreased below 10% in recent years (Kohut et al., 2012). This trend not only undermines the assumption that respondents form a random and representative sample of the population but also increases the cost and time of conducting surveys. At the same time, the Internet makes it possible to quickly collect responses from non-representative samples at virtually no cost. For example, an opt-in poll on the Xbox gaming platform collected a total of 750,148 responses during the 45 days preceding the 2012 US presidential election (Wang et al., 2015).

Such non-representative samples can provide useful information about public opinion or election outcomes if one can account for systematic differences between the survey respondents and the target population (e.g., age, gender, education, ideology, party affiliation, etc.). For example, Wang et al. (2015) used multilevel regression and poststratification to align their highly non-representative sample with the target population. They then calculated forecasts by projecting the adjusted polling results to election day, using an approach similar to Erikson and Wlezien (2008). The resulting forecasts performed equally well than traditional representative polls. This approach of forecasting from non-representative polls is highly promising as a quick and cost-effective alternative to traditional methods. However, one limitation is that the approach requires good data, which may not always be easy to obtain. For example, the first step, poststratification, requires sufficient demographical data on both the survey respondents and the target population. The second step of translating raw polling results to election forecasts requires polling data on historical elections, which may be unavailable for small-scale, local elections.

The present study tests an alternative approach for forecasting elections from non-representative surveys, which does not require additional data. Rather than utilizing responses to the traditional vote intention question, forecasts are derived from responses to the vote expectation question, which asks respondents how they expect the election to turn out. The expectation question is usually kept simple by framing the election outcome as a selection problem. While the exact phrasing depends on the specifics of the particular electoral system, citizens are commonly asked to predict the candidate (or party) that will lead the government after the election. For example, the question in the American National Election Studies (ANES) asks respondents which candidate they expect to be elected president or who will win the election in their home state. The question in the British General Election Studies asks which party will get the most MPs or, alternatively, which party will win. The question in the German Longitudinal Election Study asks which coalition of parties will form a government.

Although the use of the expectation question in pre-election surveys goes back before the emergence of intention polling (Hayes, 1936), scholars have only recently begun to study its value for predicting election outcomes in plurality elections in the UK and the US (Graefe, 2014, Graefe, 2015a, Lewis-Beck and Stegmaier, 2011, Murr, 2011, Murr, 2015a, Murr, 2015b, Rothschild and Wolfers, 2012). For example, one study compared the accuracy of the expectation question to polls, prediction markets, quantitative models, and expert judgment for predicting election winners and vote shares in the seven US presidential elections from 1988 to 2012. Across the last 100 days preceding each election, responses to the expectation question correctly predicted the election winner with a hit rate of 92%, which was more accurate than the corresponding hit rate of polls (79% correct), prediction markets (79%), expert judgment (66%), and quantitative models (86%). When predicting vote shares, expectations were again most accurate. Compared to traditional polls, expectations reduced forecast error by 51%. Compared to prediction markets, the most accurate of the four methods, error was reduced by 6% (Graefe, 2014). Another study used ANES data from the 15 elections from 1952 to 2008 to analyze the relative accuracy of the expectation question and the intention question when both are asked in the same survey. The expectation question provided more accurate forecasts than the intention question when predicting election winners, vote shares, and probabilities of victory. Furthermore, the study showed that the expectation question also performs well with small and highly non-representative samples. Aggregated and statistically adjusted expectations from two subsamples that contained only Democratic or only Republican voters yielded more accurate forecasts than the complete sample of vote intentions (Rothschild and Wolfers, 2012).

In sum, prior research has shown that expectation surveys provide highly accurate forecasts in plurality elections. However, it is unclear whether the findings generalize to elections held in more complex electoral systems. The present study thus tests the predictive value of expectations derived from a non-representative sample for forecasting an election in a multi-party system with proportional representation, namely the 2013 German Federal Election.