Evaluating Prediction Mechanisms: A Profitability Test

18 Pages Posted: 5 Mar 2021 Last revised: 25 Mar 2024

See all articles by Rajiv Sethi

Rajiv Sethi

Barnard College, Columbia University; Santa Fe Institute

Julie Seager

Columbia University, Barnard College - Department of Economics

Daniel M. Benjamin

Nova Southeastern University; USC Information Sciences Institute

Fred Morstatter

USC Information Sciences Institute

Olivia Bobrownicki

Columbia University - Barnard College

Emily Cai

Columbia University - The Fu Foundation School of Engineering and Applied Science

Yuqi Cheng

Columbia University, Barnard College - Department of Economics

Anushka Kumar

Columbia University - Barnard College

Anusha Wanganoo

Columbia University, Barnard College - Department of Economics

Date Written: January 25, 2022

Abstract

Any forecasting model can be represented by a virtual trader endowed with a budget, risk preferences, and beliefs inherited from the model. We propose and implement a profitability test for the evaluation of forecasting models based on this idea. The virtual trader enters a position and adjusts its portfolio over time in response to changes in the model forecast and prediction market prices, and its eventual profitability can be used as a measure of model accuracy. We implement this test using probabilistic forecasts for battleground states in the 2020 US presidential election and congressional elections in 2020 and 2022, using daily data from three sources: model-based forecasts published by The Economist and FiveThirtyEight, and prices from the PredictIt exchange. The proposed approach can be applied more generally to any forecasting activity, as long as models and markets referencing the same events exist.

Keywords: Prediction Markets, Structural Models, Forecast Evaluation, Presidential Elections

JEL Classification: D83, D84, G13

Suggested Citation

Sethi, Rajiv and Seager, Julie and Benjamin, Daniel and Morstatter, Fred and Bobrownicki, Olivia and Cai, Emily and Cheng, Yuqi and Kumar, Anushka and Wanganoo, Anusha, Evaluating Prediction Mechanisms: A Profitability Test (January 25, 2022). Available at SSRN: https://ssrn.com/abstract=3767544 or http://dx.doi.org/10.2139/ssrn.3767544

Rajiv Sethi (Contact Author)

Barnard College, Columbia University ( email )

3009 Broadway
New York, NY 10027
United States
212-854-5140 (Phone)

HOME PAGE: http://www.columbia.edu/~rs328/

Santa Fe Institute

1399 Hyde Park Road
Santa Fe, NM 87501
United States

Julie Seager

Columbia University, Barnard College - Department of Economics ( email )

3009 Broadway
New York, NY 10027
United States

Daniel Benjamin

Nova Southeastern University ( email )

Ft. Lauderdale, FL 33314
United States
9542625012 (Phone)

USC Information Sciences Institute ( email )

4676 Admiralty Way
Suite 1001
Marina del Rey, CA 90292
United States

Fred Morstatter

USC Information Sciences Institute ( email )

4676 Admiralty Way
Suite 1001
Marina del Rey, CA 90292
United States

Olivia Bobrownicki

Columbia University - Barnard College ( email )

3009 Broadway
New York, NY 10027
United States

Emily Cai

Columbia University - The Fu Foundation School of Engineering and Applied Science ( email )

New York, NY 10027
United States

Yuqi Cheng

Columbia University, Barnard College - Department of Economics ( email )

5281 Atlschul, 3009 Broadway
New York, NY 10027
United States

Anushka Kumar

Columbia University - Barnard College ( email )

3009 Broadway
New York, NY 10027
United States

Anusha Wanganoo

Columbia University, Barnard College - Department of Economics ( email )

3009 Broadway
New York, NY 10027
United States

Do you have negative results from your research you’d like to share?

Paper statistics

Downloads
351
Abstract Views
1,762
Rank
156,411
PlumX Metrics