Providing choice of feedback affects perceived choice but does not affect performance

Experimental approach

The study was designed and conducted on a cloud-based experimental software (www.gorilla.sc; Anwyl-Irvine et al., 2020). We aimed at examining whether allowing participants to choose the feedback frequency they receive – feedback after each trial or summary feedback after a block of 24 trials, would lead to improved performance (i.e., faster reaction time (RT) and higher accuracy) in an alternate task-switching task compared with participants who received feedback without the ability to choose. To answer this question in a laboratory-based experiment, we would have to allocate the participants to two groups: a choice group and a no-choice group. For every participant who chooses the type of feedback in the choice group, the same feedback is given without choice to one of the participants in the no-choice group. However, in an online study this is not always possible.

Therefore, participants were first allocated automatically into three experimental groups: (a) a summary feedback group – feedback after each block of trials, (b) an online feedback group – feedback after each trial (thumbs up or down), and (c) a choice group – a choice of either summary or online feedback. These three groups allowed us to create the two required groups: (a) a choice group (which was the original group) and a no-choice group (equal in numbers to the choice group and based on participants from the original online feedback and summary feedback groups).

Pre-registration and raw data repository

The study was pre-registered on aspredicted.org (https://aspredicted.org/d4gv3.pdf). Analyses that were not pre-registered are reported in the “exploratory analysis” section. The raw dataset is available on OSF (https://osf.io/p5wgx/?view_only=0ca32fa378874982b6ca28df88604413).

Participants

We recruited participants on Prolific (www.prolific.co), an online platform that allows individuals with access to the Internet to participate in online studies. The participants were paid 2.5 British Pounds for their participation. The participants were between the ages of 18–35, were fluent in English, and reported no chronic illnesses.

We used G * Power (Faul et al., 2007) to perform a priori power analysis for a two-way analysis of variance [ANOVA; group (choice/no-choice) × Test (pre/post)] comparing RTs. The effect size for this analysis was based on a recent study that examined the provision of choice in similar online tasks (Ziv & Lidor, 2021). This estimation showed an expected effect size of η²_p = 0.13 corresponding to Cohen’s f = 0.38 suggesting that participants in the choice group will show faster RTs compared to the participants in the no-choice group. Because small studies tend to have larger effect sizes (the Winner’s Curse; Button et al., 2013), we decided to reduce the effect size by 20% and used Cohen’s f = 0.30. We entered this effect size into the power analysis with the following parameters: alpha (two-sided) = 0.05, power = 0.90, and correlation among repeated measures = 0.5. The power analysis suggested that 90 participants (45 for the choice group and 45 for the no-choice group) were required to detect differences between groups with 85% power.

To account for possible dropouts, we started with a total of 151 volunteers who completed the experiment. Based on our pre-registration exclusion criteria for participation, three participants were removed from the study. This left us with 148 participants who were randomly assigned by the software to three groups: (a) choice group (n = 49, 36 females, mean age = 25.4 ± 4.4 years); (b) online feedback group (n = 51, 34 females, mean age = 27.2 ± 4.7 years); and (c) summary feedback group (n = 48, 29 females, mean age = 26.8 ± 4.8 years). From those three groups, we then created two groups: choice group (n = 49, 24 chose to receive summary feedback and 25 chose to receive online feedback), and no-choice group (n = 49, the first 24 participants from the summary feedback group and the first 25 participants from the online feedback group).

All participants completed an electronic informed consent form on the study’s website prior to participation. If participants chose not to participate, the software forwarded them to a thank you screen, and they were excused from the study. The study was approved by the Ethics Committee of the Academic College at Wingate (approval # 308).

Task

In this task, a blue or green rectangle or square were shown at the upper part or the lower part of the computer screen. When the shape was located at the upper part of the screen, the participant was required to press “f” if the shape was blue and to press “j” if the shape was green. However, when the shape was located at the bottom part of the screen, the participant was required to press “f” if the shape was square, and “j” if it was rectangle. The stimuli were presented with no time limit until the participants pressed a key. The presentation of the shapes at the upper or lower parts of the screen alternated every two shapes.

Procedure

Data collection was done remotely as each participant participated on their own computer. After reading the informed consent form and agreeing to participate in the study, all participants performed eight familiarization trials of the task. In these trials, both online feedback (a thumbs-up or thumbs-down image appeared after each trial) and summary feedback (number and percentage of correct trials) were provided. Then, the participants performed a pre-test of 24 trials with no feedback. This was followed by a practice session that included five blocks of 24 trials, where feedback was provided based on group affiliation. The participants in the online feedback group saw a thumbs-up symbol or a thumbs-down symbol for a correct or an incorrect key press, respectively, after each trial but did not receive summary feedback. The participants in the summary feedback group, on the other hand, only received information about the number of correct responses and percentage of correct responses after each of the five blocks of 24 trials (e.g., “You scored 21 out of 24. That’s 88% correct”) but did not receive online feedback. The participants in the choice group were asked whether they wanted to receive online or summary feedback and received the appropriate type of feedback based on their choice. After the five practice blocks were completed, the participants performed a post-test of 24 trials with no feedback. Finally, participants were asked to rate whether they believed that they had a choice over the type of feedback they received on a scale of 1 (not at all) to 10 (very much), and were then asked to complete the short form of the International Positive and Negative Affect Schedule (PANAS). This questionnaire was selected because it was developed in order to provide valid assessment of affect cross-culturally, and therefore is suitable for an online study where participants are from different countries and/or cultures (Thompson, 2007). The questionnaire is composed of 10 items that are rated on a scale of 1 (not at all) to 5 (very much): Determined, attentive, alert, inspired, active, afraid, nervous, upset, ashamed, and hostile. After completing this questionnaire, the participants were thanked for their participation and the study ended.

Data analyses

In our pre-registration, we decided to exclude mean RTs of under 300 ms or over 2,500 ms. This happened five times in the pre-test and seven times during practice. We also decided to exclude blocks with 13 or less correct responses out of 24. This occurred nine times in the pre-test, three times during the practice stage, and once in the post-test. Finally, we excluded participants who had at least two blocks during practice with 13 or less correct responses. This led to the exclusion of three participants from the study.

Normality was assessed using Kurtosis and Skewness values (values <2 were considered acceptable). For RT, this was mostly the case (except for Kurtosis values of 4.06, 3.73, and 3.24 for Blocks 3, 4, and 5, respectively, during practice), and therefore we used parametric statistics to analyze these data. For the number of correct responses, this was not the case (except for the pre-test) and therefore non-parametric statistics were used. We conducted a two-way ANOVA (Group (choice/no-choice) × Test (pre/post)) to assess differences in RTs. In addition, we conducted a two-way ANOVA (Group (choice/no-choice) × Block (1–5)) to assess performance during practice, and a three-way ANOVA (Group (online feedback/summary feedback) × Switch (yes/no) × Test (pre/post)) to assess differences in the switch or the no-switch trials only. Bonferroni post-hoc analyses and 95% confidence intervals were used for post-hoc testing. For analyzing the correct responses we used the Mann-Whitney test. Analyses were conducted using SPSS (version 25; IBM, Armonk, NY, USA) and the significance level was set at alpha = 0.05.

Manipulation check - perception of choice

The participants in the choice group rated their perception of choice on a scale of 1 to 10 higher (8.41 ± 2.42) than the participants in the no-choice group (5.47 ± 3.37) during the experiment, t(96) = 4.96, p < 0.001, Cohen’s d = 1.00.

Differences between pre- to post-test

RT

RTs for the pre-test and the post-test for both choice and no-choice groups are presented in Fig. 1. A two-way ANOVA (Group (choice/no-choice) × Test (pre/post)) with repeated measures on the Test factor revealed a Test effect, F(1, 90) = 32.45, p < 0.001, η²_p = 0.27. The RT in the pre-test (1,265.54 ± 395.00 ms) was longer than the RT in the post-test (1,047.42 ± 259.22 ms). There was no Group effect, F(1, 90) = 0.06, p = 0.81, η²_p = 0.00, and no interaction, F(1, 90) = 0.06, p = 0.81, η²_p = 0.00.

Differences between practice blocks

Differences between types of feedback

PANAS scores

There were no differences between the choice group and the no-choice group in any of the 10 attributes in the PANAS questionnaire (all p values > 0.09). There were also no differences between these attributes when examining all three original groups (i.e., choice, online feedback, and summary feedback; all p values > 0.4).

Exploratory analysis

Study limitations

This study has a number of limitations. First, we calculated our sample size based on a previous study that used the same tasks (e.g., Ziv & Lidor, 2021), and to be cautious reduced the effect size by 20%. This led us to a Cohen’s d of 0.6. However, a recent meta-analysis on autonomy support found Hedge’s g of 0.54 for the published literature on self-controlled learning (see McKay et al., 2021). It is then possible that our sample size was not large enough to detect this smaller effect size.

Second, in such an online study, it is possible that at least some of the participants did not pay attention to the task. However, the mean RTs in the current study (approximately between 1,000 to 1,400 ms) are similar to mean RTs reported in similar tasks performed under laboratory conditions. Salthouse et al. (2000), for example, reported values of over 1,000 ms in a switch trail making test. Minear & Shah (2008) reported RTs for three types of switching tasks. For one task, in the first block of practice, RTs were 889 and 1,027 ms for the no-switch and switch trials, respectively, and for the last block of practice these values were 697 and 763 ms. For a second task, in the first block of practice, RTs were 926 and 1,333 ms for the no-switch and switch trials, respectively, and for the last block of practice these values were 714 and 973 ms. For a third task, in the first block of practice, RTs were 1,020 and 1,395 ms for the no-switch and switch trials, respectively, and for the last block of practice these values were 740 and 1,011 ms.

In addition, as pre-registered, we removed from the study the RTs of participants who had less than 13 correct responses in a block of 24 trials because such results suggest that the participant was not attentive to the task. In future studies, researchers can use manipulation checks (see, for example, Oppenheimer, Meyvis & Davidenko, 2009). Another possible solution for participants’ attentiveness to the task in online studies is using a within-group design. Such a design can shed more light on each individual participant’s change in performance, and therefore allow more control over varying motivation and attentiveness between participants. It should be noted though that studies that compare performance in computerized tasks between laboratory and remote participation can be useful. A few studies have reported such comparisons for various aspects of online studies and their results mostly suggest that online studies can provide reliable data (e.g., Dandurand, Shultz & Onishi, 2008; Gould et al., 2015). More specifically, Crump, McDonnell & Gureckis (2013) showed similar results of tasks such as Stroop, task-switching costs, the Flanker test, and the Simon test when performed in the laboratory or online.

One final limitation is missing a control condition of a simpler RT task that requires the use of the same two keyboard keys (e.g., a choice-RT task). The differences between the RTs of such a control task and our main switching task could have shed more light on the difficulty and the level of attention required for this task. However, one previous online study required participants to perform a choice-RT task and a Simon task (a choice-RT task that requires inhibition, and therefore is more difficult to perform) with the same two keyboard keys used in the current study and showed the expected differences in RTs in both tasks (choice RT < Simon RT) (Ziv & Lidor, 2021). In addition, Crump, McDonnell & Gureckis (2013) study showed that task-switching costs could be found in online studies, similarly to laboratory-based studies. Therefore, we believe that the observations in the current online study represent an accurate assessment of participants’ performance.