An evolutionary explanation of assassins and zealots in peer review

Abstract

The peer review system aims to be effective in separating unacceptable from acceptable manuscripts. However, a reviewer can distinguish them or not. If reviewers distinguish unacceptable from acceptable manuscripts they use a fine partition of categories. But, if reviewers do not distinguish them they use a coarse partition in the evaluation of manuscripts. Most reviewers learned how to evaluate a manuscript from good and bad experiences, and they have been characterized as zealots (who uncritically favor a manuscript), assassins (who advise rejection much more frequently than the norm), and mainstream referees. In this paper we use the quasi-species model to describe the evolution of recommendation profiles in peer review. A recommendation profile is composed of a reviewer recommendation for each manuscript category under a particular categorization of manuscripts (fine or coarse). We see the reviewer mind as being built up with recommendation profiles. Assassins, zealots and mainstream reviewers are “ecologically” interrelated species whose progeny tend to mutate through errors made in the process of reviewer training. We define the recommendation profile as replicator, and selection arises because different types of recommendation profiles tend to replicate at different rates. Our results help to explain why assassins and zealots evolutionary appear in peer review because of the evolutionary success of reviewers who do not distinguish acceptable and unacceptable manuscripts.

Appendix A: Proof

We have to prove that there is an error threshold ${\hat{\epsilon }}$ decreasing in $|f -1/2|$ such that whenever errors in the reviewer training are sufficiently frequent ($\epsilon > {\hat{\epsilon }}$), then the coarse partition (under which reviewers do not distinguish categories of unacceptable and acceptable manuscripts) yields higher average reward for a population of reviewers than the fine partition.

To this aim we follow the proof of result 1 in Mengel (2012). So, given the average reward of the reviewers’ population using the coarse partition $K_\mathrm{C}$, ${\hat{\pi }} (K_\mathrm{C})$, and that using the fine partition $K_\mathrm{F}$, ${\hat{\pi }} (K_\mathrm{F})$, we show that ${\hat{\pi }} (K_\mathrm{F}) - {\hat{\pi }} (K_\mathrm{C})$ decreases in $\epsilon$ for all $\epsilon < 1/2$.

The average reward of the reviewers’ population using the partition $K_\mathrm{C}$ is the largest eigenvalue of the matrix

$$\begin{aligned} W(K_\mathrm{C}) = \left( \begin{array}{cc} (1-\epsilon ) f &{} \epsilon f \\ \epsilon (1-f) &{} \ \ (1-\epsilon )(1-f) \end{array} \right) \end{aligned}$$

which is given by

$$\begin{aligned} {\hat{\pi }} (K_\mathrm{C}) = \frac{1-\epsilon }{2} + \frac{1}{2}\sqrt{(1 - 2f)^2 - \epsilon (2 - 8f + 8f^2) + \epsilon ^2} \end{aligned}$$

and therefore, taking derivatives we find

$$\begin{aligned} \frac{\partial {\hat{\pi }} (K_\mathrm{C})}{\partial \epsilon } = \frac{1}{2} \left( \frac{ \epsilon - (1 - 2f)^2 }{ \sqrt{(1 - 2f)^2 - 2 \epsilon (1 - 2f)^2 + \epsilon ^2}} - 1 \right) \end{aligned}$$

hence, $-1 \le \frac{\partial {\hat{\pi }} (K_\mathrm{C})}{\partial \epsilon } \le 0.$

Similarly, the average reward of the reviewers’ population using the partition $K_\mathrm{F}$ is the largest eigenvalue of the matrix

$$\begin{aligned} W(K_\mathrm{F}) = \left( \begin{array}{cccc} (1-3\epsilon ) f &{} \epsilon f &{} \epsilon f &{} \epsilon f \\ \epsilon &{} (1-3\epsilon ) &{} \epsilon &{} \epsilon \\ 0 &{} 0 &{} 0 &{} 0 \\ \epsilon (1-f) &{} \epsilon (1-f) &{} \epsilon (1-f) &{} (1- 3\epsilon ) (1-f) \end{array} \right) \end{aligned}$$

which solves the equilibrium of the quasi-species equations

$$\begin{aligned} p (K_\mathrm{F}) W(K_\mathrm{F}) = {\hat{\pi }} (K_\mathrm{F}) p(K_\mathrm{F}). \end{aligned}$$

From this equilibrium we get

$$\begin{aligned} p_{(2)} ({\hat{\pi }} (K_\mathrm{F}) - (1-4\epsilon )) = p_{(4)} ({\hat{\pi }} (K_\mathrm{F}) - (1-4\epsilon )(1-f)) \end{aligned}$$

where we denote by $p_{(i)}$ the frequency of recommendation profile (i) in the population of peer reviewers using partition $K_\mathrm{F}$, and there are four possible recommendation profiles, i.e., (1) = (reject, reject); (2) = (reject, accept); (3) = (accept, reject); (4)= (accept, accept). Therefore it follows that

$$\begin{aligned} {\hat{\pi }} (K_\mathrm{F}) = \frac{ (1-4\epsilon ) ( p_{(2)} - (1-f) p_{(4)}) }{ p_{(2)} - p_{(4)} } \end{aligned}$$

We observe that taking differences between ${\hat{\pi }} (K_\mathrm{F})$ and ${\hat{\pi }} (K_\mathrm{C})$ we find (with $f \not = 0.5$)

$$\begin{aligned} {\hat{\pi }} (K_\mathrm{F}) - {\hat{\pi }} (K_\mathrm{C}) = \left\{ \begin{array}{cc} (1-f) > 0 &{} \text{ at } \epsilon = 0 \\ < 0 &{} \text{ at } \epsilon = \frac{1}{4} \end{array} \right\} \end{aligned}$$

Now taking derivatives in ${\hat{\pi }} (K_\mathrm{F})$ we get

$$\begin{aligned} \frac{\partial {\hat{\pi }} (K_\mathrm{F})}{\partial \epsilon } = \frac{ \left( \begin{array}{c} -4(p_{(2)} - p_{(4)}) \left[ ( p_{(2)} - (1-f) p_{(4)}) + \epsilon \left( \frac{\partial p_{(2)}}{\partial \epsilon } - (1-f) \frac{\partial p_{(4)}}{\partial \epsilon } \right) \right] \\ -(1-4\epsilon ) \left( \frac{\partial ( p_{(2)} - p_{(4)})}{\partial \epsilon } ( p_{(2)} - (1-f) p_{(4)}) \right) \end{array} \right) }{ (p_{(2)} - p_{(4)})^2} \end{aligned}$$

Therefore, taking differences between $\frac{\partial {\hat{\pi }} (K_\mathrm{F})}{\partial \epsilon }$ and $\frac{\partial {\hat{\pi }} (K_\mathrm{C})}{\partial \epsilon }$ we find

$$\begin{aligned} \frac{\partial {\hat{\pi }} (K_\mathrm{F})}{\partial \epsilon } - \frac{\partial {\hat{\pi }} (K_\mathrm{C})}{\partial \epsilon } = \left( -4 \left( 1 + \epsilon \left( \frac{\partial p_{(2)}}{\partial \epsilon } - (1-f) \frac{\partial p_{(4)}}{\partial \epsilon } \right) \right) - \frac{\partial ( p_{(2)} - p_{(4)})}{\partial \epsilon } \right) -(-1) < 0 \end{aligned}$$

Hence, given f, both $\frac{\partial {\hat{\pi }} (K_\mathrm{F})}{\partial \epsilon }$ and $\frac{\partial {\hat{\pi }} (K_\mathrm{C})}{\partial \epsilon }$ are negative for all values of $\epsilon$ (and continuous). Therefore, there is a ${\hat{\epsilon }}$, with $0< {\hat{\epsilon }} < \frac{1}{4}$, such that

$$\begin{aligned} {\hat{\pi }} (K_\mathrm{F}) < {\hat{\pi }} (K_\mathrm{C}) , \text{ for } \text{ all } \epsilon > {\hat{\epsilon }}. \end{aligned}$$

Also, by Lemma 2 in Mengel (2012), we have that, for any $\epsilon >0$, ${\hat{\pi }} (K_\mathrm{F}) - {\hat{\pi }} (K_\mathrm{C})$ is maximized at $f = 1/2$. Hence, following Mengel (2012), the upper bound on ${\hat{\epsilon }}(f)$ can be found by looking at the uniform case. Therefore, the set of the eigenvalues for $W(K_\mathrm{F})$ is

$$\begin{aligned} \left\{ 0, \frac{1}{2}(1-4\epsilon ), \frac{1}{4}(3-8\epsilon \pm \sqrt{1- 8\epsilon + 32 \epsilon ^2}) \right\} \end{aligned}$$

To complete the proof we only have to observe that the maximal eigenvalue for the coarse partition $W(K_\mathrm{C} )$ is given by $\lambda =1/2$ which exceeds the maximal element of the set of eigenvalues for $W(K_\mathrm{F})$, as $\epsilon > 1/4$.