Evaluating Prediction Mechanisms: A Profitability Test
18 Pages Posted: 5 Mar 2021 Last revised: 25 Mar 2024
Date Written: January 25, 2022
Abstract
Any forecasting model can be represented by a virtual trader endowed with a budget, risk preferences, and beliefs inherited from the model. We propose and implement a profitability test for the evaluation of forecasting models based on this idea. The virtual trader enters a position and adjusts its portfolio over time in response to changes in the model forecast and prediction market prices, and its eventual profitability can be used as a measure of model accuracy. We implement this test using probabilistic forecasts for battleground states in the 2020 US presidential election and congressional elections in 2020 and 2022, using daily data from three sources: model-based forecasts published by The Economist and FiveThirtyEight, and prices from the PredictIt exchange. The proposed approach can be applied more generally to any forecasting activity, as long as models and markets referencing the same events exist.
Keywords: Prediction Markets, Structural Models, Forecast Evaluation, Presidential Elections
JEL Classification: D83, D84, G13
Suggested Citation: Suggested Citation