Improving decisions with market information: an experiment on corporate prediction markets

Abstract

We conduct a lab experiment to investigate an important corporate prediction market setting: A manager needs information about the state of a project, which workers have, in order to make a state-dependent decision. Workers can potentially reveal this information by trading in a corporate prediction market. We test two different market designs to determine which provides more information to the manager and leads to better decisions. We also investigate the effect of top-down advice from the market designer to participants on how the prediction market is intended to function. Our results show that the theoretically superior market design performs worse in the lab—in terms of manager decisions—without top-down advice. With advice, manager decisions improve and both market designs perform similarly well, although the theoretically superior market design features less mis-pricing. We provide a behavioral explanation for the failure of the theoretical predictions and discuss implications for corporate prediction markets in the field.

1 Introduction

Many important corporate decisions depend on information about future conditions. Whether to fund the development of a new technology, whether to assign more resources to a project, or how many inputs to order for production: The answers to these questions depend on uncertain variables like the future demand of the technology, future problems with the project, and future sales. Thus, better information about these future conditions can help improve decision-making.

Prediction markets are one forecasting mechanism that has received a lot of attention in predicting public events like election outcomes (e.g., Forsythe et al. 1992; Wolfers and Zitzewitz 2004). A typical market trades an asset that pays off 1 if event A occurs in the future, and 0 otherwise. Traders can either buy or sell these assets. If many traders believe that A will occur, their trading will drive up the price of the asset, which creates a “wisdom of crowds” prediction of event A’s probability. Since money is at stake, these markets provide incentives to acquire and reveal information (e.g., Page and Siemroth 2017). Moreover, the logic of the efficient market hypothesis suggests that these markets aggregate information that is dispersed among traders. Indeed, prediction markets have been shown to outperform both polls (e.g., Berg et al. 2008) and experts (e.g., Spann and Skiera 2009).

Hence, prediction markets could be a valuable tool in corporations and organizations, and more generally, policy makers and decision makers might improve real (non-financial) decisions based on information inferred from asset prices. For example, central banks might glean information about economic fundamentals or inflation expectations from financial markets (e.g., Bernanke and Woodford 1997), and bond prices might reveal information about bank health to regulators and help them to audit or intervene (e.g., Prescott 2012; Sundaresan and Wang 2015). But theoretical models show that not every asset or every market is expected to be equally useful in providing pertinent information for decision-making: In some situations, the incentives of traders and decision makers are not aligned, whereas in others they are. This presents a market or asset design problem, and in this paper we test two market designs experimentally in a corporate prediction market setting to see which helps to improve a binary state-dependent decision.

Some firms are already using prediction markets, and several software companies now offer corporate prediction market packages as a service to clients.¹ Still, research on corporate prediction markets is relatively scarce. Cowgill and Zitzewitz (2015) analyzed corporate prediction markets in three large corporations and found that they outperformed expert forecasts that had been previously used as a basis for decisions. In this project, rather than analyzing existing designs, we instead ask how such markets should be designed. To the best of our knowledge, we are the first to investigate how prediction markets inform decision making, whereas previous papers almost exclusively studied forecasting quality. For the more general problem of decision makers using information from financial market prices, the literature is almost exclusively theoretical, so we provide one of the first tests of the theory and how it can be used for market design.

In our corporate framing, a manager needs information about whether a project deadline will be met in order to decide whether to invest additional resources into the project at a cost. Workers on the project are informed of the state of this project, but managers are not. One interpretation is that programmers on the project are intimately aware of the progress and potential problems, whereas the manager is not. If the state of the project is bad, then the project will miss the deadline unless it receives additional resources. If the state is good, then the deadline will be met even without additional resources. Since the optimal manager decision depends on the state of the project, we can test different prediction market designs—where workers trade and potentially reveal their information to managers—to see which leads to better decisions.

The straightforward market design that is already in use (e.g., Cowgill and Zitzewitz 2015) trades an asset that pays off 1 if the deadline is met and 0 otherwise (the “deadline asset”). However, Siemroth (2020) showed that traders have poor incentives to reveal their information in this design. If workers bet on a missed deadline in the bad state, then this drives down prices and signals to the manager to assign additional resources, which in turn leads to the deadline being met and traders losing money. Thus, workers are punished for revealing their information. In technical terms, there is no equilibrium where workers reveal their information. Intuitively, workers predicting a missed deadline suffer from a self-defeating prophecy, as their prediction triggers a manager reaction that falsifies the prediction.

An alternative market design addressing these incentive problems uses an asset that pays off 1 if the manager assigns additional resources, and 0 if not (the “action asset”, Siemroth 2020). Here, a revealing equilibrium exists: Workers trade at high prices in the bad state (suggesting to invest additional resources) and at low prices in the good state (suggesting not to invest), and the manager follows these “market recommendations.” Thus, this alternative market design aligns incentives between workers and managers. However, this design also features multiple equilibria and only one of them corresponds to the positive outcome where information is revealed.

In a lab experiment, we test these two market designs against each other in terms of managerial decision quality. We also run supporting experiments that investigate whether a fictitious “CEO message” can improve information transmission and decision-making. The CEO message explains the purpose of the prediction markets and describes a revealing trading strategy. Hence, it can be viewed as offering advice to subjects on how to act. The inclusion of the advice treatments was motivated by a sense that the interaction between the prediction market and the manager’s decision is sufficiently subtle that subjects might benefit from a description of how these markets can plausibly operate. Moreover, the CEO message can help to facilitate coordination in the financial markets based on the action asset, which feature multiple equilibria.

Based on our $2\times 2$ experimental design (varying the market design and the presence of the CEO message), our first main finding is that the theoretically superior market design actually leads to worse manager decisions in the absence of the CEO message. This is because workers play non-equilibrium strategies in the markets with the deadline asset, which leads to better manager decisions than predicted by theory. Furthermore, workers play the non-revealing equilibrium in markets with the action asset, which leads to worse decisions than if play corresponded to the fully revealing equilibrium. Our explanation for worker/trader behavior in these two treatments is based on incorrect beliefs: Their behavior is consistent with the mistaken belief that managers do not react to the information contained in prediction market prices. In the language of Eyster et al. (2019), workers appear to incorrectly believe that managers are cursed, i.e., that managers are unable to infer the informational content of prices. Our behavioral explanation can account for all observed qualitative results and exactly predicts the average price levels in three out of four state-treatment cells.

Our second main finding is that the CEO message—advice to use a revealing trading strategy—significantly improves both the information revealed by the prediction markets and managerial decisions by up to 25 percentage points. With the CEO message, managerial decisions between the two market designs are no longer significantly different, although the point estimate on correct managerial decisions is 9 percentage points higher with the action asset. A secondary criterion favoring the action asset design is that it exhibits significantly less mis-pricing, which should make it more stable. Thus, managerial decision quality being roughly equal, a practical recommendation would be to use the action asset design with top-down advice akin to our CEO message.

Our findings have at least two important implications. First, absent advice, subjects are unable to realize the theoretically superior outcome of the action asset market design. This suggests that a theory based on rational expectations is not well-suited for mechanism design with inexperienced players in this complex setting. Even in the last few rounds of the experiment, after allowing for experience and potential learning, subjects do not converge towards the theoretical predictions. Our behavioral explanation suggests that incorrect beliefs among traders drive this result, which can potentially be corrected via communication by the market designer.

Second, while the treatments without the CEO message provide valuable insights into strategic reasoning in interactive market settings, these conditions are unlikely to prevail in the field. If a firm implements a corporate prediction market to aid in managerial decisions, then it would also explicitly communicate the purpose of the market and explain how it is meant to be used. This is precisely what our CEO message does, and we show it improves outcomes significantly. In Sect. 5, we also discuss some methodological implications for experimental economists.

We have a few remarks about our design choices. First, we study a setting in which the state of the project is common knowledge among workers, rather than a more general setting in which workers have asymmetric information. However, the ability of financial markets to at least partially aggregate private information has been extensively researched and is well-documented in the experimental literature. Moreover, it is not the focus of this paper. For these reasons, we chose the simplest possible information structure that still allows us to shed light on our main research question: whether asset prices can be used to improve decision making, when these decisions in turn feed back into asset values. Second, if we take our setting literally, it is reasonable to ask whether a prediction market is even necessary: In other words, can the manager learn the state of the project simply by asking any individual worker? In many relevant cases, we believe the answer is no. An individual worker might be reluctant to reveal the state of the project to the manager, for a variety of reasons outside of both the theoretical model and the experiment. For instance, workers might fear that the manager will “shoot the messenger” for delivering bad news. A prediction market mitigates this problem by offering anonymity to participants, with the additional benefit of providing financial incentives for information revelation (unlike the flat incentives that arise when the manager just asks an individual worker). Finally, our experiment controls for other forces that might also be at play in workplace environments. This allows for a cleaner and more direct test of the underlying theory. We further discuss this issue in Sect. 5.

Overall, the potential of prediction markets to improve decisions is often mentioned but rarely investigated or explicitly allowed in experimental settings. Hence, our main contributions are to show how prediction markets can aid decision making and to demonstrate that the choice of market design very much affects information revelation and subsequent decision quality. To the best of our knowledge, we are the first to experimentally study prediction market design for the purpose of improving organizational decisions. In addition, our investigation of top-down advice in a complex market signaling game is novel and valuable for corporate applications.

1.1 Literature

While the framing of a project deadline we consider here may appear somewhat specific, our setting gives insights into the more general problem of a decision-maker having to make a state-dependent decision, which is informed by traders in a financial market. This possibility has been considered in several theoretical papers in other contexts, for example managers learning from financial market prices to decide upon investment opportunities (e.g., Bond et al. 2010; Bond and Goldstein 2015; Edmans et al. 2015; Dow et al. 2017) or central banks learning from financial market prices to decide upon monetary policy (e.g., Bernanke and Woodford 1997). Indeed, though the framing differs, our setting is formally very close to Bernanke and Woodford (1997)’s static model when considering only two states and two decisions, so the problem we investigate is of interest beyond corporate prediction markets.

In the prediction market literature, several studies have investigated the informational efficiency of these markets in non-corporate settings, such as Oliven and Rietz (2004), Berg et al. (2008), Page and Clemen (2013) based on field data or Page and Siemroth (2017, 2019), Corgnet et al. (2018, 2019), Choo et al. (2019) based on experimental data. The evidence on markets incorporating public information is positive, although the experimental work shows that there is typically under-reaction to private information (the latter paper being an exception). The experimental literature has also investigated potential problems of prediction markets, such as manipulation, where the evidence is mixed (Hanson et al. 2006; Deck et al. 2013). Moreover, thin markets with very few traders were found to hamper information aggregation (Healy et al. 2010). Ledyard et al. (2009) investigate prediction markets when the number of variables to be predicted is large, and find that performance suffers in this case.

Studies with field data from corporate prediction markets are very rare. Cowgill and Zitzewitz (2015) provide analyses of markets from three different major companies and find that they improve upon the tools used before. However, they also identify an optimism bias where assets based on events that would be positive for the company are overpriced. Plott and Chen (2002) use corporate prediction markets to forecast sales. They find the market forecasts are an improvement over the existing forecasting methods. Gillen et al. (2017) use a parimutuel betting rather than a prediction market mechanism to forecast sales in a company, and find that the mechanism outperforms the existing forecasting techniques.

There are very few studies of settings where markets can affect non-market decisions. Kogan et al. (2011) consider a setting where subjects in a group choose an effort level in a coordination game. A pre-play prediction market could potentially help to facilitate coordination by letting subjects bet on the effort levels in the subsequent coordination game. However, they find that prediction markets actually decrease effort levels in the coordination game and lead to worse outcomes. Our setting differs, because in our case the market can potentially reveal information about a state variable that the decision maker needs but does not have. Moreover, we consider a concrete corporate setting where markets are already used, whereas Kogan et al. (2011) consider an abstract setting.

Davis et al. (2014) investigate whether financial markets can help a regulator make a conversion decision for contingent convertible bonds. They consider a prediction market as an additional tool to provide information and find that the treatment with the additional prediction market does best. In contrast to our setting, the prediction market is only a minor addition in one treatment that coexists with the market for contingent convertibles, which is the focus of their study.

2 Theory and predictions

2.1 Model set-up

This section introduces a simple setting whose qualitative predictions we test in the lab, but many features of the model such as information structures, state spaces etc. can be generalized (see Siemroth 2020).

Players A corporate prediction market consists of $N\ge 2$ competitive and risk neutral workers/traders indexed by $i=1,\ldots , N$, and a risk neutral manager/decision maker. Workers are endowed with a budget of $0<w<\infty$ in cash and $e>0$ units of the risky asset.

State of the project and information structure The state of the project that workers are working on is binary with $\theta \in \{0,1\}$, where $\theta =1$ is the good state and $\theta =0$ is the bad state. The prior probability is ${\text{ Pr }}(\theta =1)=0.5$, i.e., the state of the project is good half of the time. All workers observe the realization of $\theta$, i.e., know whether the project is in a good state or in a bad state, but the manager does not. The idea is that workers on the project have detailed information about the progress and potential problems (e.g., programmers being aware of unanticipated bugs), whereas the manager does not directly work on the project and hence does not have knowledge of the state of the project.

Project deadline Whether the deadline for the project will be reached depends on two components, the state of the project $\theta$ and a manager decision $d\in \{0,1\}$. In the experimental frame, the manager decision $d=1$ means the manager assigns more resources to the project at a cost and $d=0$ means she does not. The project deadline is reached if $\theta =1$ or $d=1$ or both, i.e., if the state is good or more resources are assigned to the project. The project deadline is missed if and only if $\theta =0$ and $d=0$.

Assets Traders can hold a risk-less asset with return normalized to zero (cash) and trade a risky asset. In the currently practiced form of deadline prediction markets, the risky asset pays 1 if and only if the deadline is met, and 0 if the deadline is missed (see, e.g., Cowgill and Zitzewitz 2015). We call this asset design the deadline asset. Formally, let the value of the deadline asset be $a(\theta ,d)=\varvec{1}\{d=1\vee \theta =1\}$, where $\varvec{1}\{.\}$ is the indicator function.

In another treatment we use another risky asset. The action asset has a value of 1 if the manager assigns additional resources to the project ($d=1$) and a value of 0 if she does not ($d=0$). Formally, $a(\theta ,d)=d$. The prediction market either trades the deadline asset or the action asset, not both. That is, we are comparing markets with the deadline asset to markets with the action asset.

Timing At $t=0$ nature draws $\theta$ and workers observe it. At $t=1$ workers trade in a competitive prediction market yielding a market clearing price $p\in [0,1]$. At $t=2$ the manager can observe the prediction market price p, and make a decision based on (the information contained in) the price p, d(p). At $t=3$, assets pay out and payoffs realize. This timing is displayed in Fig. 1.

Fig. 1

Timeline with workers and manager

Payoffs Workers receive a bonus $b_w\ge 0$ if the deadline is reached. Moreover, workers keep their trading profits. They submit net demand functions $x_i(p)$, so their trading profit at price p is $[a(\theta ,d)-p]x_i(p)$ (where $x_i(p)<0$ is a sale of the asset). Managers receive a bonus $b_m>0$ if the deadline is reached. The managerial cost of assigning additional resources is $c>0$ with $b_m>c$.

Overall, the crucial feature in this model is that managers do not know the state of the project $\theta$, but would like to know it, because if the state is good then they can save the cost of assigning additional resources. In the bad state managers want to assign additional resources, since it is necessary to reach the deadline and the benefit outweighs the cost ($b_m>c$). Because workers know $\theta$, they can potentially reveal that information to the manager via trading prices. This is exactly the idea of prediction markets informing real (i.e., non-financial) decisions.

Our equilibrium concept is the standard one for competitive financial markets with asymmetric information, except there is an additional player (the manager) who optimally reacts to whatever information is revealed by prices, and this reaction (in particular its impact on asset values) is correctly anticipated by traders. For a similar definition see Bond et al. (2010).

Definition 1

A rational expectations equilibrium (REE) with manager consists of

• (i) Optimal trader net demands for the risky asset given their information and the information revealed in prices,

  $$\begin{aligned} & x_i(p,\theta )=\arg \max _x \mathbb {E}[x(a(\theta ,d(p))-p)|p(\theta )=p,\theta ]~\forall i \\& {\text{ s.t. }} xp\le w,~x\ge -e \end{aligned}$$

• (ii) an optimal manager decision function d(p) given the information revealed in prices,

  $$\begin{aligned} d(p)=\arg \max _{d\in \{0,1\}} \mathbb {E}_\theta [\varvec{1}\{d=1\vee \theta =1\}b_m-dc|p(\theta )=p], \end{aligned}$$

• (iii) and a price function $p(\theta )$ that clears the market for every $\theta \in \{0,1\}$, i.e.,

  $$\begin{aligned} \sum _{i=1}^N x_i(p=p(\theta ),\theta )=0 ~\forall \theta . \end{aligned}$$

A fully revealing equilibrium is an equilibrium with $p(\theta =0)\ne p(\theta =1)$, i.e., where market clearing prices differ by state, so that the manager can infer the state. A non-revealing equilibrium is an equilibrium with $p(\theta =0)=p(\theta =1)$.

2.2 Analysis and predictions

The main question is whether the incentives are aligned in such a corporate prediction market so that workers trade in a way that reveals their information when they correctly anticipate the manager reaction to prices.² That is, we ask whether there is a fully revealing equilibrium. Such an equilibrium might not exist, because the manager reacts to trading prices, so traders may inadvertently be punished for revealing their information to the manager. More generally, an equilibrium non-existence can arise because asset values are endogenous to a third party’s decision that reacts to prices (e.g., Bernanke and Woodford 1997; Bond et al. 2010; Siemroth 2019).

The following propositions determine whether revealing equilibria exist. The proofs are in the online appendix.

Proposition 1

(Deadline asset)

• (i) In a corporate prediction market with deadline asset, there exists no fully revealing equilibrium.

• (ii) A non-revealing equilibrium exists if and only if $b_m/2 \ge c$. In this equilibrium, the manager always assigns additional resources, the deadline is always met, and the prediction market always predicts that the deadline will be met $(p(\theta )=1~\forall \theta )$.

In the experiment we chose parameters such that $b_m/2=c$, hence a non-revealing equilibrium exists but no fully revealing one. Thus, the theoretical prediction for prediction markets with deadline assets is not good: This asset design means that information revelation by traders is not incentive compatible.

The reason is quite simple: In a fully revealing equilibrium, prices have to differ by state for information revelation to occur, but due to the manager decision the asset value would be the same in all states, as the deadline is always met. In a competitive market, this cannot be an equilibrium outcome: Prices have to equal (expected) asset values.

Specifically, traders want to trade at a price less than 1 only if the asset value is less than 1. Clearly, in the good state that can never be the case as the deadline is always reached. But potentially the asset value can be less than 1 in the bad state. So suppose traders always trade at $p=1$ in the good state and at $p<1$ in the bad state, making market prices fully revealing: The manager can invert the price function and infer the underlying state of the project. This means that the manager always assigns additional resources in the bad state but not in the good one, so the deadline is met even in the bad state. But this cannot be an equilibrium, because for a price $p<1$ and an asset value of 1, every trader wants to buy assets, but then markets cannot clear and prices would have to adjust upwards in the bad state, until prices are no longer revealing.

Proposition 2

(Action asset)

• (i) In a corporate prediction market with action asset, there exists a fully revealing equilibrium with $p(\theta =1)=0$ and $p(\theta =0)=1$.

• (ii) A non-revealing pure strategy equilibrium exists. If $b_m/2 \ge c$, then the non-revealing equilibrium is $p(\theta )=1~\forall \theta$ and the manager always assigns additional resources. If $b_m/2 < c$, then the non-revealing equilibrium is $p(\theta )=0~\forall \theta$ and the manager never assigns additional resources.

• (iii) A continuum of non-revealing mixed strategy equilibria exists if $b_m/2=c$. In these equilibria, prices are state-independent with $p(\theta )=\alpha ~\forall \theta$ with $\alpha \in [0,1]$, and the manager assigns additional resources with probability $\alpha$.

The action asset improves the incentives for information revelation, because a fully revealing equilibrium exists. The action asset pays off a positive amount if the manager assigns additional resources. Thus, unlike with the deadline asset, the asset value does not depend on the outcome (whether or not the deadline is reached) which the manager seeks to manipulate.

The fully revealing equilibrium with the action asset can be interpreted as a recommendation by traders to the manager: A price of $p=1$ in the bad state predicts that the manager assigns additional resources, and a price of $p=0$ in the good state predicts the manager does not assign additional resources. Because the manager only wants to assign additional resources in the bad state, it is in her own best interest to follow the “recommendation” by the market: Additional resources if and only if the state is bad, i.e., if and only if $p=1$. Thus, this recommendation is self-fulfilling, and this is why it is an equilibrium.

But the action asset also has an undesirable feature: equilibrium multiplicity. While the fully revealing equilibrium always exists, so does at least one non-revealing equilibrium. Nevertheless, the theory favors the action asset with a fully revealing equilibrium over the deadline asset where no fully revealing equilibrium exists. Thus, for all parameter values, the theoretical prediction is that the action asset should lead to better manager decisions, because the traders have incentives to reveal their information via trading, unlike in the case of the deadline asset.

Prediction 1

Existence of fully revealing equilibrium: The action asset leads to (i) more accurate manager beliefs about the state and to (ii) better manager decisions compared to the deadline asset.

Moreover, if equilibrium multiplicity and coordination on an equilibrium are a problem, then we might expect there to be more variation in prices with the action asset. This is an important consideration for corporate prediction markets, as more variation in prices makes the information contained in prices harder to interpret.

Prediction 2

Equilibrium multiplicity: The action asset leads to larger variance of prices compared to the deadline asset.

When taking the theory to the lab, we also run supporting experiments in which a fictitious “CEO message” is provided to the subjects and read aloud before the start of the session. The purpose of the CEO message is to provide additional context for the prediction market and to make non-binding suggestions to the subjects for how to behave. Notably, these suggestions are different for the two asset market designs.³ Our decision to include a CEO message in both treatments is motivated by the desire to more carefully sort out alternative explanations for why the CEO message might be effective. In the action asset market, the CEO message provides a recommendation that is consistent with the fully revealing equilibrium. In the deadline asset market, the CEO message provides a recommendation that is not consistent with any equilibrium. If subjects implement the recommended strategies outlined in the CEO message for both market designs, then we have evidence in favor of the “experimenter demand effect” (i.e., subjects respond to advice per se). If subjects implement the recommended strategies outlined in the CEO message only for the action asset market, then we have evidence in favor of the “equilibrium effect” (i.e., subjects respond to self-enforcing advice).

Our intuition is that subjects will begin the experiment by implementing a version of the recommended strategies in both treatments. In that case, the equilibrium recommendations (for the action asset) will persist while the non-equilibrium recommendations (for the deadline asset) will provide incentives for subjects to deviate to other strategies. Indeed, this basic logic has been confirmed by experimental studies that investigate the effectiveness of recommendations for mediating equilibrium selection in coordination games (e.g., Brandts and MacLeod 1995; Van Huyck et al. 1992). This allows us to make the following prediction:

Prediction 3

The CEO message improves manager decisions for the action asset, but not for the deadline asset.

3 Experimental design

Table 1 provides a summary of our experimental design. We use a $2 \times 2$ experimental design that varies (1) whether the asset pays out depending on the deadline or the manager action and (2) whether an asset-specific CEO message is included, resulting in a total of four treatments. Each session of each treatment consists of either one or two cohorts, where a cohort is a fixed group of 12 subjects.⁴ Each cohort participates in 11 rounds (1 practice round and 10 rounds that count), where each round corresponds to a single play of a prediction market that mirrors the model with the timing from Fig. 1.⁵

We now describe the general structure of the experiment. Payoffs in the experiment are expressed in tokens, where 6 tokens = £1. Before the start of the experiment, experimental instructions are handed out to the subjects and read aloud.⁶ Subjects have the opportunity to ask any clarification questions, which are answered publicly. In the message treatments (DC and AC), a fictitious CEO message—which explains the purpose of the prediction markets—is then handed out to subjects and also read aloud (see the online appendix for the exact wording). In the no-message treatments (DN and AN), this message is omitted.

At the beginning of the experiment, each subject is randomly assigned to the role of either a worker or a manager. Each cohort of 12 subjects consists of 8 workers and 4 managers. Subjects’ roles are fixed across all rounds of the experiment.

In each round, each cohort faces a project deadline. The feasibility of meeting the project deadline depends on the state of the project, which is randomly determined to be either “good” or “bad” with equal probability. Workers, but not managers, are informed of the state of the project. If the state of the project is good, then the project deadline is always met. If the state of the project is bad, then the project deadline is only met if the manager decides to invest additional resources into the project.

After the state of the project is realized, each worker receives a token endowment of 40 tokens and an asset endowment of 4 units. The asset payoff depends on the treatment. In the deadline asset treatments (DN and DC), each unit of the asset pays 10 tokens if the project deadline is met, and 0 tokens otherwise. In the action asset treatments (AN and AC), each unit of the asset pays 10 tokens if the manager decides to invest additional resources into the project, and 0 tokens otherwise. In the remainder of the paper, we rescale the asset values (and prices) from [0, 10] to [0, 1] to match the usual prediction market practice and to ease notation.

Workers participate in a prediction market organized as a standard continuous double auction, where they are free to offer to buy/sell units of the asset for a price of their choosing (create limit orders) or to accept the offers of others (execute market orders). Thus, the prediction market either predicts whether the deadline will be met (DN and DC) or whether additional resources will be assigned by the manager (AN and AC). Managers cannot trade, but they can observe the trading of the workers and, in particular, the trading prices. Thus, the prediction market can potentially inform managers. Trading closes after two minutes.

After the prediction market closes, each manager receives a trading summary consisting of a list of all trading prices as well as the average trading price. Each manager then completes two tasks: She decides whether to invest additional resources into the project (at a cost) and she reports her belief about the likelihood that the state of the project is good.⁷

At the end of each round, payoffs are realized and reported to subjects on their computer screens. For each manager, one of her two tasks is randomly selected to determine her round payoff. If the investment decision is selected, the manager earns a salary of 30 tokens, receives a bonus of 60 tokens if the project deadline is met, and incurs a cost of 30 tokens if she decided to invest additional resources into the project. If the belief elicitation is selected, then the manager earns $90\cdot \big (1 - (1 - \frac{r}{100})^2 \big )$ tokens if the state of the project is good and $90\cdot \big (1 - (\frac{r}{100})^2\big )$ tokens if the state of the project is bad, where $r \in [0, 100]$ represents the manager’s reported probability belief that the state is good.

For each cohort of 12 subjects (8 workers and 4 managers), the investment decision of one manager is randomly selected to determine workers’ payoffs. Having four managers in every prediction market, instead of just one, allows us to increase statistical power by generating four manager decisions per round. Workers’ payoffs are determined based on their final portfolio holdings (in tokens and assets). For example, a worker who does not trade has 4 assets and 40 tokens at the end of the round, which can yield a maximum payoff of 80 tokens. In addition, each worker receives a bonus of 10 tokens if the project deadline is met. Thus, 90 tokens is the maximum average payoff of workers, which mirrors the maximum payoff of managers for either task.

At the end of the experiment, we elicit subjects’ risk preferences using the procedure of Gneezy and Potters (1997).⁸$^{,}$⁹ Subjects’ final earnings are the sum of their payoff from one randomly selected round (out of the 10 rounds that count), their payoff from the risk elicitation task, and a £5 show-up payment.

Table 1 A summary of our experimental treatments

3.1 Implementation

The experiment was run at ESSEXLab at the University of Essex from October 2018 to December 2018. Subjects were recruited via email invitation from the student population. A total of 288 subjects participated in the experiment. Average subject earnings were £19.52 (including a £5 show-up payment), with a minimum payment of £7 and a maximum payment of £41. The experiment was programmed and conducted with the software z-Tree (Fischbacher 2007).

4 Results

4.1 No-message treatments

Before going into detailed analyses of the treatment effects, we display summary statistics for the main outcome variables in Table 2. The first three variables relate to manager outcomes, while the last two describe market behavior. The first variable uses the responses from the incentivized manager belief elicitation about the good state. CorrectBelief equals 1 if managers put 100% on the realized state of the project, equals 0 if managers put 100% on the wrong state, and generally equals $b\in [0,1]$ if managers put $100\cdot b$% on the correct state.

The variable CorrectDecision takes a value of 1 if the manager makes the correct state-dependent decision (i.e., invest additional resources if and only if the state is bad), and a value of 0 otherwise. The variable ManagerPayoff is the payoff of a manager from her decisions in a round, which can be between 30 and 90, depending on the state and manager decisions.

The price variables refer to the prices of transactions, and we separately summarize these transaction prices for markets with a bad state of the project, and for a good state of the project. This allows us to determine whether prices are informative about the state.

Table 2 Summary statistics of all outcome variables

4.1.1 Manager outcomes

We start with the analysis of the no-message treatments. Since we are primarily interested in learning how prediction markets can improve real decisions, the most important outcome measure is whether managers make correct choices (CorrectDecision). Table 3 displays OLS estimates (linear probability model) in column 3.¹⁰

With the action asset, manager decisions are 12.1 percentage points less likely to be correct, compared to the deadline asset design. This is a statistically significant difference, and of large magnitude. Given a base rate of 66.2% correct decisions with the deadline asset, managers barely perform better than 50%, which a coin flipping manager would achieve, or one who always assigns additional resources independent of the state. This result is surprising, as it is a highly significant result in the opposite direction of what the theory predicts. Despite the action asset having a fully revealing equilibrium, which would allow managers to achieve 100% correct decisions, managers actually perform worse compared to the deadline asset where no fully revealing equilibrium exists.

Table 3 No-message treatments: manager outcomes

Since we also elicited manager beliefs about the state, we can dig deeper and analyze whether the worse manager performance is due to worse decisions or due to less information from the prediction market.¹¹ Recall that we elicited how likely managers think the good state is given the prediction market prices they observed, which was incentivized, and leads to variable CorrectBelief which states how much probability mass managers put on the correct state.

Column 1 in Table 3 displays the belief results. The direction of the treatment effect is the same as for manager decisions, and the magnitude is very similar. Managers in markets with the action asset put 11.2 percentage points less on the correct state of the project. This is very clear and direct evidence that the prediction market with the action asset does not reveal as much information about the state of the project, compared to the deadline asset. Indeed, similar to the decisions, managers put barely more than 50% on the correct state, compared to 63.1% in markets with the deadline asset. Thus, it is not that managers with the action asset make systematically worse decisions while having the same information. The problem is that the prediction market with the action asset does not provide as much information about the state.

In column 5 of Table 3, we also analyze the manager action payoff, which is 90 if the deadline is reached without additional resources, 60 if the deadline is reached with additional resources, and 30 if the deadline is not reached. The manager payoff is a measure of the value or welfare that is created by the manager decision (which in turn depends on the market information). In line with the two other manager outcome measures, outcomes are significantly worse with the action asset.¹² In other words, we find significant evidence against Prediction 1 (i.e., the prediction of better performance of the action asset). This is summarized by our first main result:

Result 1

Without messages, the action asset leads to worse outcomes. Markets with the action asset provide less information, lead to fewer correct manager decisions, and lead to lower manager payoffs.

Since the prediction market and manager interaction is a complex one, it is worth looking into potential learning effects. Perhaps the action asset performs better in the later rounds, suggesting it might work better with experience. Columns 2 and 4 in Table 3 estimate a linear trend over the 10 rounds of the experiment for manager beliefs and correct decisions. There is a significant improvement over time for the deadline asset, with CorrectBelief increasing by 2.2 percentage points per round on average and correct manager decisions increasing by 3.8 percentage points per round. However, as the ActionAsset $\times$ Round interaction shows, this learning effect is significantly smaller with the action asset. Indeed, based on the estimated linear trends, there is no significant improvement over time with the action asset.¹³ Thus, while the two asset designs are not significantly different initially, the action asset falls behind over time.

Result 2

Without messages, there is significant improvement over time due to learning with the deadline asset, but no significant improvement with the action asset.

4.1.2 Prediction market prices

We can formally analyze the prices we observe in the prediction market and see how they differ by asset design. In particular, revealing or separating equilibria rely on the fact that the observable actions—here, the prices that are determined by the worker trades—differ by state. Hence, we can formally analyze the degree of separation of prices by state. Columns 1 and 2 in Table 4 do exactly that.

Table 4 No-message treatments: market data

For the deadline asset in column 1, the average transaction prices¹⁴ are about 9.5 percentage points¹⁵ larger in the good state compared to the bad state. This difference is statistically significant, so we have clear separation of prices by state. For the action asset in column 2, we do not observe significant separation: Average transaction prices are the same in both states. And if we include group fixed effects to account for the fact that the state-price mapping might be different in different groups (not displayed), then these conclusions follow as well. Hence, it is clearly visible in the raw price data that there is more separation with the deadline asset, and so it is no surprise that managers learn more from the prices of the deadline asset. The online appendix confirms this finding when using transaction prices rather than mean prices as signals of the state.¹⁶

In column 3-4, we also tested whether transaction volume differs by state in any of the market designs. Since managers see all transaction prices, they also see the trading volume. Hence, if volume differs by state, then it could reveal information even if mean prices do not. We find that there is no significant difference in volume by state in either of the market designs. We can therefore rule out that trading volume revealed more information in the action asset design; if anything, trading volume also favors the deadline asset design, giving us a consistent picture why manager decisions are better with the deadline asset.

Column 5 in Table 4 tests whether the standard deviation of prices in a market vary by asset design, in a regression where data is aggregated at the market (round) level. We find no significant difference. Hence, the difficulty in extracting information from prediction market prices is not due to more noise or variance in one market design. Thus, we find significant evidence against Prediction 2 (i.e., the prediction of larger variance of prices with the action asset). This is summarized by the following result:

Result 3

Without messages, markets with the deadline asset have a significant difference in transaction prices between the good and the bad states, i.e., prices reveal information about the state. There is no significant difference in prices between states in markets with the action asset. Moreover, there is no significant difference in the variance of prices between the two asset designs.

Fig. 2

Average transaction prices over time by state (top row), and correct manager beliefs over time (bottom row)

In order to illustrate the developments over time better, we graph the average transaction price by state over the 10 rounds of the experiment, and the resulting probabilities that managers put on the correct state over time, in Fig. 2. The conclusions on treatment differences from the regression tables, which pool all rounds, are consistent with these graphs.

First, in the top left, for the deadline asset treatment, the average transaction prices show noticeable differences by state, i.e., prices in the good state tend to be larger from round 2 onwards, and appear to increase in the later rounds. In contrast, the top right figure shows the average transaction price by state for the action asset treatment, and these do not differ noticeably in any of the rounds. While there is a clear time trend in the absolute price level—prices tend to decrease over time—this trend is present in both states, hence it does not make prices more or less informative about the state over time.

Second, the bottom left graphs the probability that managers put on the correct state in the deadline asset treatment. This correct belief tends to increase over time—in line with the significantly positive time trend in the regression in Table 3—and in line with the separation in prices displayed above. Hence, it is clearly visible that separating prices help managers infer the underlying state of the project. In contrast, in the bottom right, the correct beliefs for managers do have a clear linear trend over time in the action asset treatment, and the correct beliefs in the last two rounds are not significantly different from the correct beliefs in the first round. And for no round can we reject the hypothesis that beliefs are more than 50% correct in this treatment. Our straightforward explanation is that prices are not an informative signal about the underlying state in this treatment, hence managers cannot do better than random guessing.

4.2 Explanation of behavior in the no-message treatments

4.2.1 What is to be explained

In the deadline asset treatments, prices are higher in the good state (on average 0.65) than in the bad state (on average 0.55), which reveals some information about the state. Because of this, manager decisions are significantly better than random guessing or probability matching the prior: The correct state dependent decision is taken in 66% of cases. These outcomes are not equilibrium outcomes.

In the action asset treatments, average prices are the same in both states (at about 0.48). Thus, there is no separation and no information revelation, and manager decisions reflect this: The correct state dependent decision is taken in 54% of cases, which is not statistically different from random guessing (which would imply 50% correct decisions). These outcomes are equilibrium outcomes in the mixed-strategy equilibrium where the manager assigns additional resources in 50% of cases and workers trade at prices of $p=0.5$.

Thus, our findings are not completely explained by the rational expectations model in Sect. 2, as the deadline asset performance is better than what the theory predicts. We now provide an explanation for these experimental findings based on incorrect expectations.¹⁷

4.2.2 A non-equilibrium explanation

Managers extract information from prices in the deadline asset treatments where we do observe significantly different prices by state, and use that information to improve decisions above what they could achieve by just using the prior distribution. Managers cannot extract information in the action asset treatments, because prices do not reveal any (as mean prices do not differ by state). Hence, the evidence supports the idea that managers are using the information contained in prices, as would be expected in a rational expectations equilibrium. For an explanation we instead look toward the workers who determine prices.

We need beliefs or a decision heuristic for the traders that explains revealing prices with one asset but not the other, and also explains equilibrium outcomes with one asset but not the other. Our goal is to find a parsimonious explanation, and it focuses on the beliefs of the workers.

Suppose workers believe that managers will randomly invest additional resources with 50% probability, independently of what the workers themselves are doing (i.e., independent of prices). Such beliefs are not consistent with our evidence, but could plausibly arise in at least two different ways (though our explanation does not depend on how these beliefs arise).

First, such beliefs are predicted by level-k models from behavioral game theory. This class of models relaxes the equilibrium assumption that players’ beliefs are consistent with other players’ actions. In particular, level-k models posit the existence of non-strategic L0 players (who randomize uniformly across all available actions) and strategic Lk players, $k = 1,2,\dots , \infty$ (who play a best-response to the belief that all other players are Lk−1). The empirical content of level-k models and the frequencies of various types can then be estimated using laboratory data. Broadly speaking, experimental evidence suggests that the largest fraction of subjects are type $\textit{L1}$.¹⁸ In our setting, L1 workers would then play a best-response to the belief that managers are L0, i.e., to the belief that managers will invest with 50% probability.

Second, workers could believe that managers are “cursed” in the sense of Eyster et al. (2019). Intuitively, cursed agents in financial markets either neglect or fail to fully take into account the informational content of prices. In the extreme case where a manager makes no inference from asset trading prices, she would instead rely on the prior distribution of states and hence may just randomize in her resource decision. Thus, workers’ (mistaken) perception that they are interacting with cursed managers could also generate the belief that managers will invest additional resources with 50% probability.¹⁹

What outcomes would we observe if workers held such beliefs and best responded to them? Given such beliefs, workers anticipate an expected asset value of 0.5 in the action asset treatments, because the asset value is completely determined by the manager action, who according to their beliefs is randomizing. The market clearing price given such beliefs is 0.5 in both states. This is what we observe in the data.

The deadline asset is different because its value directly depends both on the manager action and on the state. Specifically, if the state is good, then the asset value is 1 no matter the manager decision. But if the state is bad, then the asset value is 1 only if the manager assigns additional resources, and 0 otherwise. Thus, given the worker beliefs, their expected asset value is lower in the bad state, because the asset pays off only if the manager assigns additional resources, which they believe she does in 50% of the cases. Consequently, the market clearing price in the bad state is 0.5, which is what we observe (the mean price is 0.55), and higher in the good state, which we also observe.

Thus, our incorrect expectations account of worker behavior can explain the observed mean prices in both states in the action asset treatments exactly, and the mean prices in the bad state in the deadline asset treatment exactly. It also explains why there is separation with one asset but not the other, and by extension, non-equilibrium outcomes with one asset but not the other. However, it does not explain the exact price level in the good state in the deadline asset treatment: Workers should be trading at a price of 1 whereas they trade at an average price of 0.65. Thus, our explanation accurately explains average prices in three out of four state-treatment cells, and all qualitative state-differences by treatment.²⁰

Why do these incorrect expectations among workers lead to an equilibrium outcome in the action asset treatment but not in the deadline asset treatment? The reason is that these beliefs are self-fulfilling and thus correct in the former, but not the latter. In the action asset treatment, if workers believe the manager to randomize no matter what they do, then they should only be trading at prices of 0.5 in both states. If they do that, then prices do not reveal anything about the state and the manager can indeed rationally randomize (i.e., mix her strategies). Thus, these myopic beliefs can lead to a non-revealing equilibrium outcome, though clearly not to the also existing fully revealing equilibrium, which would require a belief that managers actually react to the information contained in prices.

Our explanation suggests a remedy for this problem. If traders incorrectly believe that managers do not react to prices, then it should be possible to correct the trader beliefs by providing an example of a revealing trading strategy as well as the appropriate manager reaction. If this is done publicly, such that the information is common knowledge among all participants, then traders will know that managers know a strategy profile where the manager reacts to prices. In the following treatments, our CEO message provides such a public announcement.

4.3 CEO message treatments

We now analyze the CEO message treatments. The CEO message explains the general idea behind the prediction market (i.e., to reveal worker information to managers for better decisions) and provides an example of a fully revealing trading strategy. While the basic structure of the CEO message is identical for both asset market designs, the details of the described trading strategies differ.²¹ For the action asset, the CEO message describes a fully revealing trading strategy that implements the fully revealing equilibrium. For the deadline asset, the CEO message describes a fully revealing trading strategy that is not consistent with any equilibrium. Neither of the two CEO messages claim that the described strategies are optimal, nor do they explicitly recommend the use of these strategies. Rather, the CEO messages are framed as the “vision” of the CEO for how the prediction markets can plausibly operate. It is emphasized that subjects are free to trade in any fashion.

The inferiority of the action asset in the no-message treatments may have at least two different causes. First, the equilibrium may have been too complex for subjects to identify on their own through introspection. Second, subjects may not have been able to coordinate on the fully revealing equilibrium.²² The CEO message can potentially mitigate both of these problems. First, it raises awareness of a trading strategy that workers may not have noticed (which is part of an equilibrium for the action asset). Second, because it is a public message, it facilitates coordination of the financial market and it helps to establish a possible mapping from states to prices. This latter point is important for managers to be able to infer the state from observed prices.

4.3.1 Manager outcomes

Columns 1–3 in Table 5 show the same manager outcomes for the CEO message treatments that we analyzed for the no-message treatments before. On all three outcome measures, there is no significant difference between markets using the action and deadline assets. Furthermore, the point estimates of the difference are also relatively small. A notable exception (in terms of magnitude) is the share of correct manager decisions, which is 9.2 percentage points larger with the action asset. This estimate, while not significantly different from zero, is an indicator of the potential of the action asset.

Overall, while the CEO message does not make the action asset perform better as predicted by theory, it does remove the negative difference found in the no-message treatments. And while there is significant learning over the 10 rounds of the experiment, such that manager decisions are better in later rounds, we found no significantly different learning between the two asset designs (learning regressions are not displayed).

Result 4

In the CEO message treatments, there is no significant difference in outcomes or learning between the action and deadline assets.

Columns 6–7 in Table 5 compare action asset markets with and without the CEO message. The CEO message significantly improves outcomes across all measures. This difference is also large in magnitude: It improves correct decisions from barely better than a coin flip to about 79% correct decisions. Columns 4–5 show that the CEO message also significantly improves manager beliefs in the deadline asset markets, whereas the share of correct decisions has a positive but insignificant point estimate. Overall, the magnitudes of improvement are lower for the deadline asset since it started from a higher base rate in the no-message treatments. In other words, we find significant evidence in favor of Prediction 3 (i.e., the prediction that the CEO message improves manager decisions for the action asset but not for the deadline asset). This is summarized by the following result:

Result 5

The CEO message improves outcomes for markets with the action asset. It also improves manager beliefs in deadline asset markets, but not decisions.

Table 5 Effects of CEO message: Manager outcomes

Our treatments can at least partially pin down the underlying mechanism for why the CEO message improves outcomes. In particular, we are able to distinguish between two possible channels. First, if subjects respond to advice per se, then the CEO message will be effective in both market designs by inducing an experimenter demand effect. Second, if subjects only respond to self-enforcing advice, then the CEO message will be effective only in the action asset market where the described trading strategies are consistent with equilibrium behavior. Our finding that the CEO message improves manager decisions in the action asset, but not the deadline asset, casts doubt on the experimenter demand hypothesis and provides support for the equilibrium hypothesis.

4.3.2 Prediction market prices

Table 6 CEO message treatments: Market prices

Again, we can analyze the effect of the CEO message on prices, and how the message interacts with the different asset designs. Table 6 tests whether there is price separation in the CEO message treatments in columns 1 and 2. For the deadline asset, prices differ significantly between state by on average 31 percentage points. For the action asset, there is separation by on average 35 percentage points, and prices in the good state are lower, exactly as suggested in the CEO message that described the fully revealing equilibrium.²³

Columns 3 and 4 estimate the effect of the CEO message on state-price differences directly, as captured by the interaction term. For the deadline asset, the difference in average prices between the good and the bad states increases significantly due to the CEO message, i.e., separation increases. For the action asset, the difference between the good and the bad states significantly decreases. Since that difference was virtually zero without the message, separation (as the absolute value of the difference between states) also significantly increases. Thus, the CEO message causes significant changes in market behavior for both assets. In particular, it increases price separation and thereby makes prices more informative and more revealing about the underlying state.

Result 6

With the CEO message, markets with the deadline and action assets have a significant difference in transaction prices between the good and the bad states, i.e., prices reveal information about the state. There is significantly more information revelation for both assets due to the CEO message.

4.3.3 Calibration and mis-pricing

Since prediction market prices are usually interpreted as probability forecasts, it is useful to check for calibration of these prices. Formally, calibration requires prices to fulfill $p=\mathbb {E}[v|p]$, where v is the asset payoff. For example, calibration requires that for asset prices of 0.2, the asset does indeed pay off in 20% of the cases; for asset prices of 0.3, the asset pays off in 30% of the cases, and so on. Hence, calibration is a mild form of absence of mis-pricing, which is closely related to weak-form informational efficiency.

Fig. 3

Calibration of prediction market prices, by treatment

In standard prediction market settings with exogenous asset values but asymmetric information among traders about those values, market prices are typically found to be well calibrated (e.g., Page and Clemen 2013 in the field for short maturities; Dreber et al. 2015 in the field; Page and Siemroth 2017 in the lab). In our setting, where we investigate prediction markets with endogenous asset values, it is clear that prices are not well calibrated.

The calibration plots, which compare all possible prices on the x-axis and the associated expected asset values given these prices on the y-axis, are displayed in Fig. 3.²⁴ We use transaction-level prices in this analysis, with standard errors clustered at market level.

According to Fig. 3, the calibration curve is almost flat, especially in the no-message treatments. By contrast, perfect calibration would require the curve to be on the 45 degree line, so that conditional expected asset values match prices. The CEO message improves calibration, especially for the action asset. Indeed, the action asset with CEO message produces the best calibration, although even here the calibration curve significantly differs from the 45 degree line for most prices. This is further evidence that endogenous asset values are challenging for subjects and markets, and simple forms of mis-pricing can arise.

To formally test whether there are differences in mis-pricing by treatment, we can compute the absolute difference between the price and the ex post asset value (zero or one) for each transaction.²⁵ We refer to this mis-pricing measure as “price error.” Treatments where prices always equal asset values would have a price error of zero. Treatments where prices on average differ more from asset values would have larger price errors.

Table 7 Mispricing

Table 7 shows that markets with the action asset have significantly smaller price errors, and the CEO message does not significantly reduce price errors. In particular, the action asset has 19 percentage points smaller price errors in the no-message treatments and 17 percentage points smaller price errors in the CEO message treatments. This means that workers tend to trade more at prices that diverge from the actual asset values in the deadline asset markets, so that the losses of those on the wrong end of the transaction are larger. For example, if the asset value is $a(\theta ,d)=1$, then the seller at a price $p<1$ makes a loss, while the buyer makes a profit, and the losses and profits increase with the difference between price and asset value, overall having a zero sum. In this sense, mis-pricing is closely linked to non-best responses (the seller had better not traded, or traded at a higher price), and could be viewed as a measure of regret for one party in the transaction. We would then expect a market design with lower mis-pricing to be more stable in the long term and in the field, just as we typically expect equilibrium behavior (which implies a price error of zero) to be more stable.

Result 7

Markets with the action asset are better calibrated and exhibit less mis-pricing. The CEO message does not significantly change mis-pricing.

This result is perhaps the strongest one in favour of the action asset design. And given no significant differences between manager decision quality, we could use mis-pricing as a secondary criterion to argue that—if supported by something similar to a CEO message—the action asset market design may be preferable, because some workers lose less money by revealing their information.

As a final point, the online appendix analyses the effects of asset design and CEO message on trade volume. More trading indicates higher liquidity and makes participation in these markets more attractive; a lack of liquidity is viewed as a major threat to the proper functioning of double auction prediction markets (e.g., Wolfers and Zitzewitz 2006). The regressions show that the action asset increases trading volume relative to the deadline asset, while the CEO message has no significant effect.

5 Discussion

Our results have two important implications: one for practitioners planning to use corporate prediction markets to improve decision making, and another for researchers planning to run laboratory experiments to inform practitioners.

First, even if the market/asset is properly designed so that a desirable (i.e., revealing) equilibrium exists, our results suggest that the interaction between the prediction market and the manager’s decision is perhaps too complex for users to arrive at the desirable equilibrium themselves. To alleviate this problem, firms planning to use corporate prediction markets should clearly communicate their objectives to their employees. Our CEO message implemented this top-down advice in a simple way, and the change significantly improved outcomes.

Second, our no-message treatments are conducted in line with the standard practices of experimental economics. That is, the instructions provided to subjects explain the setup, strategy spaces, and payoffs with a neutral framing and with an eye toward mitigating experimenter demand effects. Clearly, subjects failed to reach the fully revealing equilibrium in the action asset treatment and the unique non-revealing equilibrium in the deadline asset treatment. This was true even in the later rounds of the experiment, so there was no obvious pattern of learning or convergence towards the desirable equilibrium.

These findings provide valuable insights for theory. However, the no-message treatments may not be the ideal setup for generating robust and portable results that can carry over to the field. In particular, a company running a corporate prediction market will typically communicate with their employees and try to steer them towards better outcomes, perhaps even providing explicit advice like in our CEO message. Standard game-theoretic experiments (like our no-message treatments), on the other hand, are deliberately non-leading in their design in order to allow the experimenter to observe how subjects behave in strategic settings without guidance.

Thus, we argue that the CEO message treatments provide the superior test of how these markets are likely to perform outside the lab. In other words, we view the message treatments as a “policy” experiment, which tells us something about the optimal design of corporate prediction markets, while we view the no-message treatments as a “theory-testing” experiment, which tells us something about strategic reasoning and behavior in a more stylized setting. Indeed, these different justifications for conducting laboratory studies have been identified much earlier in the history of experimental economics. In the framework of Roth (1986), the message treatments advance the goal of “whispering in the ears of princes” while the no-message treatments advance the goal of “speaking to theorists.”

Similar arguments have been raised by Zizzo (2010), who points out that advice can be useful in improving the external validity of experimental results if that same advice would also have been provided in the real world setting that the experimenter wishes to mimic. Zizzo (2010) further notes that advice can be important if it is the primary object of research, i.e., if the researcher explicitly wishes to identify the effects of advice on behavior and outcomes. There is also a growing experimental literature that investigates the impact of advice to participants in strategy-proof matching mechanisms (e.g. Guillen and Hing 2014; Guillen and Hakimov 2018; Ding and Schotter 2019). These studies are “motivated by the idea that economic mechanisms should be tested in the environments in which they are used in the real world” (Ding and Schotter 2019). Our experiment continues in this tradition. Our results suggest that organizations with corporate prediction markets should be transparent in communicating to their users how they intend these markets to function.

In practical applications in the field, there may be additional challenges that we carefully controlled for in the lab. First, the state of the project was exogenous here, but in many cases the workers directly influence that state (e.g., by putting in more effort). In the worst case, this might induce additional incentives for workers to derail the project in order to make money by betting on a missed deadline. One theoretical study considers this problem (Ottaviani and Sørensen 2007), but empirical evidence is scarce. In the best case, it might induce additional incentives to work harder on the project. Second, there could be complications if the managers are allowed to trade in the prediction markets, unlike in our setting. Since manager decisions have a direct impact on the asset value—indeed, in the action asset design their action solely determines the asset value—this could induce incentives for managers to change their decisions for higher trading profit. Third, workers in companies might be afraid of getting fired if news of problems with their project gets out, which might induce additional incentives to keep bad news under wraps. This might make these markets less effective than they are in our lab setting where we removed the threat of being fired. Overall, our lab study focused on the problem of information transmission from workers to decision makers via corporate prediction markets if decisions feed back to asset values. To cleanly investigate the different designs, we deliberately shut off the potential complications just discussed, which should however be taken into account in field applications.

We would also like to discuss the use of belief elicitation in our experiment. While beliefs often play a role in economic models, they are not directly observed and so economists have developed incentivized procedures to elicit true beliefs. The effectiveness of these procedures at eliciting true beliefs is still an open question, with possible belief distortion arising from multiple sources. First, there is a concern that the act of belief elicitation itself alters beliefs compared to the beliefs that subjects would hold in the absence of belief elicitation. Second, there is experimental evidence that subjects may exhibit over-confidence and inflate their beliefs about their own abilities or other ego-relevant characteristics (Burks et al. 2013; Charness et al. 2018). Third, depending on the particular procedure that is used, risk-averse subjects may have incentives to misreport their beliefs in a direction that is closer to a uniform prior (Offerman et al. 2009). Fourth, the problem of hedging can arise if subjects are paid based on both their actions and their reported beliefs (Blanco et al. 2010). Finally, the size of financial incentives can affect the accuracy of reported beliefs (Coutts 2019; Zimmermann 2020). Despite these concerns, Schotter and Trevino (2014) argue that the practice of eliciting incentivized beliefs in the lab generates accurate and meaningful data, as measured by the consistency between subjects’ actions and their reported beliefs.

Although our main outcome variable of interest is the manager’s decision, we use the quadratic scoring rule to elicit the manager’s belief about the state of the project (which is incentive-compatible under the assumption of risk-neutrality). We argue that our elicited belief data is reliable for several reasons. First, we elicit beliefs about the realization of a binary random variable that is not ego-relevant. Second, subjects are provided detailed guidance about the belief elicitation procedure in the experimental instructions, which includes both a statement that reporting their true beliefs maximizes their expected earnings as well as example calculations to illustrate that point. Third, there is no opportunity for hedging in our experiment since the manager is randomly paid based on only one of her two tasks: either the accuracy of her reported belief or her investment decision. Finally, we find that 86% of managers’ investment decisions are a best-response to their reported beliefs about the state of the project, which compares favorably to estimates from previous experimental studies.²⁶

6 Concluding remarks

We consider a setting where a manager needs information that workers have in order to make a correct state-dependent decision about whether to invest additional resources into a project facing a deadline. Corporate prediction markets are one possible mechanism that can potentially reveal this information to managers. In a laboratory experiment, we test two different prediction market designs that may help in the managerial decision making process. Markets with the deadline asset let workers predict whether the deadline of a project will be met. This design has previously been used in the field. Markets with the action asset let workers predict whether additional resources will be assigned by the manager. Standard theory shows that the action asset provides superior incentives for workers/traders to reveal their information, whereas there is no revealing equilibrium with the deadline asset.

The results from our no-message treatments, however, show that the theoretically inferior deadline asset does better than the action asset in the lab, as measured by correct state-dependent manager decisions. This result is surprising, and suggests that the complex nature of the interaction between the prediction market and the manager’s decision is not entirely understood by all workers. Our behavioral explanation of the observed outcomes is consistent with workers/traders having incorrect beliefs about how managers react to the market. Hence, subjects fail to reach the desirable revealing equilibrium in the action asset markets, which requires traders to understand that their trades affect manager decisions.

We also run two additional treatments where both workers and managers receive a fictitious CEO message that describes the purpose of the prediction markets, and provides an example of a revealing trading strategy. With this additional message, the information revealed by the prediction market and the share of correct manager decisions increase considerably in the action asset design and, to a lesser extent, in the deadline asset design. These results suggest that firms running corporate prediction markets should describe how these markets are used for decision making, and how they are supposed to work. Not only does this help traders understand how information can be transmitted via the market, it can also help mediate equilibrium selection in the action asset markets. When comparing both market designs with the CEO message, the action asset design has lower mis-pricing, which suggests it could be more stable in the long run.

From here, there are several promising avenues for future research. An important next step is to test these market designs in the field, and over the long run. An intriguing result from our experiment is that a CEO message describing a revealing equilibrium trading strategy (action asset) produces similarly good manager decisions as a CEO message describing a revealing non-equilibrium trading strategy (deadline asset). This parity might change in the field in the long run, especially once traders recognize that they regularly lose money in the deadline asset markets when revealing their information. Another salient question is whether a corporate prediction market is actually the best mechanism for this task. While we compared outcomes under different prediction market designs in this study, mechanism design might suggest a different optimal mechanism altogether depending on the specifics of the problem. Future work might therefore compare the performance of different mechanisms along the lines of Healy et al. (2010).