Targeted testing for bias in order assignment, with an application to Texas election ballots
Introduction
Each year an unusual ritual takes place in school district offices, city halls, and county courthouses across the state of Texas: the drawing of the order in which candidates for public office will be placed on the ballot, as required by state law. Candidates often attend these drawings to ensure that the agent conducting them does not manipulate the ordering and place a competitor higher on the ballot, conferring upon them an electoral advantage known as the ballot-order effect. In Texas primary and runoff elections for statewide office (Grant, 2017) find this effect to be sizeable and monotonic in ballot order, especially in low-profile or low-information races. This finding is corroborated for other states by several studies cited therein, while Meredith and Salant (2013) obtain broadly similar results for local elections in California. The possibility that such orderings might be prejudicially biased, either consciously or unconsciously, by the agents executing them is not far-fetched. Darcy and McAllister (1990) noted the evidence for such bias in their review of the early literature on the ballot order effect.3 One can test statistically for such bias when orderings (i.e. permutations) of the same set of items are conducted repeatedly and (presumably) independently times, as in Texas, where ballot order for primary and runoff elections is determined by agents at the county level, even for offices contested statewide. A test for uniform random ordering when is elementary, but this is not the case when multiple items are being ordered.
The general problem of testing for the uniform randomness of permutations does not just arise in political science. A classic example from computer science is testing random number generators via the randomness of repeated sequences of digits (Knuth, 1981).4 In addition, when ordering contestants in musical or athletic contests, randomization helps ensure fairness because arbiters’ fastidiousness can vary over the course of a day or competition. It is desirable in sequencing courtroom trials for the same reason (Danziger et al., 2011). Most generally, the ballot order effect is an example of a more general psychological phenomenon, the primacy effect (cf. Murdock, 1962), in which the first-listed of a set of options tends to be chosen more frequently. Thus, in many scenarios in which a set of competing decisions must be made sequentially without prejudice, the agents ordering those decisions may be tempted to manipulate the orderings in accordance with their preferences. Testing for randomization should reduce the likelihood of such manipulations and can uncover them when they occur.
Despite the generality of the problem, however, a consensus on testing methodology has not emerged. Even within the ballot order literature, a variety of options are used. Grant (2017) applies Fisher’s Exact Test to the cross-tabulation of candidates and ballot positions in Texas, to determine if this cross-tabulation is likely to have occurred by random chance. Meredith and Salant (2013) apply Pearson’s chi-squared test to determine if incumbent candidates are equally likely to end up at any position on the ballot. And Ho and Imai (2008) apply a series of rank tests based on the average absolute difference in rank between pairs of letters to randomized alphabets that are used for ballot ordering in California.
Each approach has limitations. The first procedure, Fisher’s Exact Test, aggregates the observed orderings into a contingency table, where is the number of items and order positions. Among other problems,5 such aggregation loses relevant information contained in the orderings themselves. It is possible – and, in political applications, probable – that agents with opposing prejudicial biases manipulate orderings in opposite directions, but these offsetting manipulations may not be apparent in the aggregate. Similarly, important information is lost by applying Pearson’s chi-squared test to incumbents alone: all information about non-incumbents is ignored.
In addition, unless is very small or very large, these tests, along with those of Ho and Imai (2008), may lack the sensitivity needed to detect the specific deviations from uniform randomness that may be encountered. This limited sensitivity would derive, in part, from the untargeted nature of such tests, which do not utilize a priori information that could increase their power. Such information is often available in political and other applications in which human agents perform the orderings; any deviations from uniform randomness are likely to reflect these agents’ preferences.
In this paper, procedures are developed that utilize the individual orderings, not their aggregation, and that will detect departures from uniform randomness that are to be expected from a priori information about the characteristics of the items and the preference criteria of the agents executing the ordering. In Section 2, the null hypothesis of an unbiased order assignment is represented by several forms of exchangeability of a random permutation. In Section 3, the alternative hypothesis of bias in order assignment is represented by compatibility with an assumed preferential ranking (ties permitted) of the items, while in Section 4 bias is represented by linear concordance with assumed preference scores of the items. In both cases methods for detecting the corresponding alternatives are obtained. Section 5 analyzes these tests’ power relative to one another and an neutral alternative – the rank test of Ho and Imai (2008) – and outlines their practical application when the true form of deviations from uniform randomness is unknown. In Section 6 these procedures are applied to five races in the 2014 Texas Republican primary. Significant evidence of bias in at least one of the approximately 245 reporting counties is found in three of the five races; in two of these, significant evidence is found for bias in at least six and ten counties.
The tests developed in this paper rely on assumptions about agents’ preferences that, while appropriate in the political context, are somewhat strong. A sequel develops power-enhancing tests for more general sets of preferences and for the most general case of all, in which no a priori knowledge is available.
Section snippets
Unbiased and biased order assignments
Suppose that items, arbitrarily labeled , are being ordered by each of agents. An ordering is simply a permutation of such that item is assigned position , . The primacy effect implies that positions near 1 () in the ordering are advantageous, while positions near () are disadvantageous.
We shall assume that the ordering selected by an agent is a realization of a random permutation (quite possibly degenerate). The gold standard for
Preference criteria (PC) expressed by ranks
Suppose that an agent ranks the items (with ties permitted) in accordance with a particular PC, so that a low (high) rank indicates agreement (disagreement) with the PC. This yields a partitioning of the items into blocks of sizes (, ). Items in block have a lower (the same) PC ranking than (as) those in if ().
If each then the items are totally ordered w.r.t. the PC, while if one or more then they are partially ordered.9
Preference criteria (PC) expressed by scores
In Section 3 it was assumed that if an agent is unbiased, they will select a random ordering according to the exchangeable model , while if biased, they will select an ordering that conforms exactly to a partitioning or its opposite , specified by a uni-directional or bi-directional preference criterion (PC). In some cases, however, this may over-simplify the true behavioral processes that generate bias. Other processes may generate biased orderings that only partly conform with a PC, for
Power analysis and practical test usage
We now perform a simulation study to compare the statistical power of our rank-compatibility and linear concordance tests to a neutral alternative, the rank test used by Ho and Imai (2008).13 This test was chosen as the standard of comparison because it neither aggregates nor disregards information, obtains simulation-consistent size for a given level (the null distribution must be
Results from the 2014 Texas Republican primary election
Finally, we apply these three tests to ballot-order data from the 2014 Texas Republican primary elections (used in Grant, 2017) for of the 254 counties in Texas.14
Concluding remarks
The ballot-order effect is an example of a more general psychological phenomenon, the primacy effect (cf. Murdock, 1962), in which the first-listed of a set of options tends to be chosen more frequently. Other situations in which the primacy effect may affect outcomes include athletic or artistic competitions, funding approval processes – such as those at the NIH and NSF – and college admissions decisions. In such scenarios one can often identify various preference criteria – political, social,
Acknowledgments
We would like to thank our reviewer for the insightful commentary on the first submission of the paper, as well as Kosuke Imai for his feedback on an early draft of this paper. Thanks also to the Department of Defense, the National Institutes of Health, and Sam Houston State University for funding that permitted this work. And cheers to interdisciplinary collaborations!
Funding
This work was supported by the National Institutes of Health, United States of America [Grant No. HSN268201-600310A] and the
References (13)
- et al.
Ballot position effects
Elect. Stud.
(1990) - et al.
Extraneous factors in judicial decisions
Proc. Natl. Acad. Sci.
(2011) An Introduction to Probability Theory and Its Applications Vol. II
(1966)The ballot order effect is huge: evidence from Texas
Public Choice
(2017)- et al.
Estimating causal effects of ballot order from a randomized natural experimentthe California alphabet lottery, 1978–2002
Public Opin. Q.
(2008) Rank Correlation Methods
(1948)
Cited by (2)
Uncovering bias in order assignment
2023, Economic InquiryUncovering Bias in Order Assignment
2020, SSRN
- 1
Research was supported in part by National Institutes of Health, United States of America Grant HHSN268201-600310A.
- 2
Research was supported in part by U.S. Department of Defense Grant H98230-10-C-0263.