Virtual reality check: Statistical power, reported results, and the validity of research on the psychology of virtual reality and immersive environments

doi:10.1016/j.chb.2019.06.015

Computers in Human Behavior

Volume 100, November 2019, Pages 70-78

https://doi.org/10.1016/j.chb.2019.06.015 Get rights and content

Highlights

•
We conducted a systematic review of empirical research on VR.
•
We included 61 articles reporting 1122 statistical tests.
•
We evaluated data collection practices, statistical reporting, and data availability.
•
There was a high error rate in statistical reporting and overall low transparency.
•
Paths forward to advance our empirical understanding of VR effects are discussed.

Abstract

Virtual reality (VR) is a popular subject of scientific study across a variety of academic fields. In the present study we evaluate methodological trends in behavioral research on VR with respect to data collection practices, statistical reporting, and data availability. In line with this goal, we conducted a meta-scientific analysis of 61 articles encompassing a total of 1122 statistical tests and highlight three emergent trends that inform our understanding of past and future studies focused on VR. Conclusions from analysis of the data include a high incidence of errors in statistical reporting, and a general lack of transparency with respect to the availability of study data. Transparency in data analysis, increased statistical power, and more careful reporting of statistical outcomes are suggested to heighten methodological rigor and improve reproducibility in the field of VR research.

Section snippets

Declaration of competing interests

The authors report no conflicts of interests associated with this research.

Defining virtual reality: approaches and challenges

There are a variety of approaches to both conceptualizing and researching VR technology. On the industry side, the technology and entertainment non-profit Virtual Reality Society defines VR as a type of immersive and realistic emulation that “present[s] our senses with a computer generated virtual environment that we can explore in some fashion” (Virtual Reality Society, 2017a). First-generation VR researchers including Ivan Sutherland and Myron Krueger similarly defined VR through the use of

Materials and method

This study used a meta-scientific analysis to describe and analyze the methods used in VR experiments in communication and psychology research. The goal was to determine the average statistical power, distribution of reported p-values, rate of statistical reporting errors, and availability of data in these experiments. Such methods have previously been used by scholars to survey a field's existing body of knowledge for reliability, accountability, and accuracy—important values for replicability

Descriptive statistics

The screened articles (N = 61) ranged in publication year from 2002 to 2018. In total, 29 journals were represented, with the Annual Review of CyberTherapy and Telemedicine being the most frequently represented with 14 articles, followed by Media Psychology (n = 8) and Frontiers in Psychology (n = 7). At the test level (K = 1122), the majority of statistical tests came from Frontiers in Psychology (k = 167), followed by Media Psychology (k = 157) and CyberPsychology, Behavior, and Social

Discussion

In line with prior meta-scientific research on methodologies used in communication and psychology, this study found trends suggesting that there is room for improvement in methodological practice in the scientific pursuit of understanding VR's effects on users. Findings from this study speak to four main areas including statistical power, patterns of reported p-values, the frequency of reporting errors, and the availability of study materials.

Our first research question concerned the degree of

Conclusion

The four research questions pursued in this study touch on four stages of the research process: Study design, data analysis, reporting findings, and sharing knowledge. The observations we report in this study suggest there are clear methodological challenges facing scientific practice at each of these four stages. This data provides an empirically-grounded cause for alarm for social scientists studying VR and a structure for further exploration. One can only hope that these findings can be used

References (57)

A. Garcia-Palacios et al.
Virtual reality in the treatment of spider phobia: A controlled study
Behaviour Research and Therapy
(2002)
O. Baus et al.
Moving from virtual reality exposure-based therapy to augmented reality exposure-based therapy: A review
Frontiers in Human Neuroscience
(2014)
F. Biocca
Communication within virtual reality: Creating a space for research
Journal of Communication
(1992)
F. Biocca et al.
The vision of virtual reality
J. Cohen
A power primer
Psychological Bulletin
(1992)
A. Coursol et al.
Effect of positive findings on submission and acceptance rates: A note on meta-analysis bias
Professional Psychology: Research and Practice
(1986)
J.J. Cummings et al.
How immersive is enough? A meta-analysis of the effect of immersive technology on user presence
Media Psychology
(2016)
M. Elson et al.
The science of technology and human behavior: Standards, old and new
Journal of Media Psychology
(2017)
S. Epskamp et al.
statcheck: Extract statistics from articles and recompute p values
J. Fox et al.
Virtual reality: A survival guide for the social scientist
Journal of Media Psychology
(2009)

J. Fox et al.

Virtual experiences, physical behaviors: The effect of presence on imitation of an eating avatar

Presence

(2009)

Frontiers in Human Neuroscience. (n.d.) Author Guidelines. Retrieved from:...

J.J. Gagnier et al.

Misconceptions, misuses, and misinterpretations of p values and significance testing

The Journal of Bone and Joint Surgery

(2017)

A. Gelman

Why does the replication crisis seem worse in psychology? Slate

M.E. Grabe et al.

Explicating sensationalism in television news: Content and the bells and whistles of form

Journal of Broadcasting & Electronic Media

(2001)

S. Greenland et al.

European Journal of Epidemiology

(2016)

A.G. Greenwald

Consequences of prejudice against the null hypothesis

Psychological Bulletin

(1975)

L. Gregg et al.

Virtual reality in mental health: A review of the literature

Social Psychiatry and Psychiatric Epidemiology

(2007)

Y. Hu

Health communication research in the digital age: A systematic review

Journal of Communication in Healthcare

(2015)

J. Ioannidis

Why most published research findings are false

PLoS Medicine

(2005)

L.K. John et al.

Measuring the prevalence of questionable research practices with incentives for truth telling

Psychological Science

(2012)

S.S. Kalyanaraman et al.

The virtual doppelganger: Effects of a virtual reality simulator on perceptions of schizophrenia

The Journal of Nervous and Mental Disease

(2010)

K. Kelley et al.

On effect size

Psychological Methods

(2012)

R. Kirk

Practical significance: A concept whose time has come

Educational and Psychological Measurement

(1996)

E. Klinger et al.

Virtual reality therapy versus cognitive behavior therapy for social phobia: A preliminary controlled study

CyberPsychology and Behavior

(2005)

D. Lakens

The probability of p-values as a function of the statistical power of a test

J. Lanier et al.

An insider's view of the future of virtual reality

Journal of Communication

(1992)

J.M. Loomis et al.

Immersive virtual environment technology as a basic research tool in psychology

Behavior Research Methods, Instruments, & Computers

(1999)

Cited by (35)

Reducing dehumanisation through virtual reality: prospects and pitfalls
2023, Current Opinion in Behavioral Sciences
This paper presents evidence that Virtual reality (VR) has the potential to reduce dehumanisation via a range of different routes, notably the experience of being present in a situation with outgroup members, experiencing virtual contact and interaction with the outgroup and taking on the perspective or even body of an outgroup member. In addition, it hightlights key questions that require future research, including the strength of empirical evidence that VR can indeed reduce dehumanisation, the mechanisms by which such a process occurs and the ethical issues in treating VR as an 'ultimate empathy machine'.
Analyzing the potential of virtual reality-supported training for industrial assembly tasks
2023, Computers in Industry
Virtual Reality (VR) in industry offers opportunities to decrease design and production costs, maintain product quality and reduce the time needed to go from product concept to production. Due to its interactive nature, VR is also used for simulations of training sessions to communicate knowledge on assembly and maintenance processes. So far, there are hardly any scientific findings on the effectiveness of virtual learning and training for industrial applications. It is also unclear how training can be improved in virtual environments to maximize the learning outcome. This article investigates whether VR-supported training leads to an increase in learning success compared to traditional training-on-the-job accompanied by a tutor. Two variations of a VR tool for the training of a simplified assembly processes were developed. A laboratory study with 64 participants was conducted to get an insight into the importance of virtual instructions and their effect on learning success and perceived workload. The results indicate that small modifications in a VR training application have a significant effect on learning outcome. In particular, a lower level of visual guidance can cause a higher level of mental workload during the training, forcing the user to actively deal with the learning content. Based on these findings, the opportunities and risks of current VR training simulations are discussed and recommendations to increase the learning effect are formulated.
Experiencing herd immunity in virtual reality increases COVID-19 vaccination intention: Evidence from a large-scale field intervention study
2023, Computers in Human Behavior
Citation Excerpt :
Nowak et al. (2020) investigated the effect of three different modalities of communicating vaccine benefits: e-pamphlet, video, and VR on influenza vaccination intake intentions. However, none of the interventions led to a significant increase in vaccination intentions compared to the control condition, similarly to other VR studies (Lanier et al., 2019), the study was underpowered to detect more subtle effects (n = 48 in the VR condition) which are usually reported in the studies investigating vaccination communication (Freeman et al., 2021), indicating the necessity for more large-scale testing. A more recent study conducted as an online experiment (Mottelson et al., 2021) investigated whether a VR intervention can increase COVID-19 vaccination intentions in a sample of VR headset owners.
This study investigates the impact of an immersive virtual reality (VR) simulation of herd immunity on vaccination intentions and its potential underlying mechanisms. In this preregistered field study, N = 654 participants were randomly assigned to one of the three VR conditions: (1) Gamified Herd Immunity; (2) Gamified Herd Immunity + Empathy (with additional narrative elements); (3) Control (gamified with no vaccination-related content). In the Gamified Herd Immunity simulation, participants embodied a vulnerable person and navigated a wedding venue trying to avoid getting infected. A total of 455 participants with below maximum intentions to take a novel vaccine and without severe cybersickness were analyzed. The Gamified Herd Immunity + Empathy and the Gamified Herd Immunity conditions increased vaccination intentions by 6.68 and 7.06 points on a 0–100 scale, respectively, compared to 1.91 for the Control condition. The Gamified Herd Immunity + Empathy condition enhanced empathy significantly more than the Gamified Herd Immunity condition but did not result in higher vaccination intentions. Experienced presence was related to the change in vaccination intentions. The results suggest that VR vaccination communication can effectively increase vaccination intentions; the effect is not solely due to the technological novelty and does not depend on empathy.
Immersive virtual reality in the age of the Metaverse: A hybrid-narrative review based on the technology affordance perspective
2022, Journal of Strategic Information Systems
Immersive virtual reality (VR) that utilizes head-mounted displays (HMD) is one of the key emerging technologies of the 21st century and has drawn keen attention from consumers, practitioners, and scholars in various disciplines. Nevertheless, the information systems (IS) discipline has neglected immersive VR, given that only a handful of studies have been published in mainstream IS journals. However, the recent advancements in immersive VR technology provide new opportunities for organizations and IS researchers. In light of these points, we reviewed the immersive VR literature to provide a holistic view of opportunities and challenges for organizations and future research directions for the IS field. By examining the technical capabilities of immersive VR and the previous literature, we identified five affordances: embodiment, interactivity, navigability, sense-ability, and create-ability. Our review of the 151 studies from the IS and related fields synthesized how these affordances were utilized in various research domains. Guided by the affordance-actualization theory, we also identified the strategic opportunities and challenges that come with implementing VR. The actualization of immersive VR affordances in organizations is indeed a fruitful area for IS scholars as there are various venues to move the IS field as well as the VR research in the organizational context forward.
Biased by being there: The persuasive impact of spatial presence on cognitive processing
2021, Computers in Human Behavior
Immersive media forms such as virtual reality simulations are considered suitable vehicles for persuasive communication because they enhance the perception of spatial presence and non-mediation. Although the persuasive impact of spatial presence has been confirmed multiple times and immersive media have entered the mainstream market, its influence on individuals’ cognitive processing has not yet been analyzed empirically. An experimental study was conducted with a 2 × 2 between-subjects design (N = 129) and varying levels of spatial presence. The quality of persuasive arguments was manipulated to detect the dominant processing mode. In line with the bias hypothesis of the heuristic–systematic model of information processing, the results indicated that individuals who experienced high levels of spatial presence evaluated the content more positively because they used heuristic processing. The positive evaluation consequently led to biased systematic processing, resulting in persuasive effects, even when the arguments were weak. The implications of the results are discussed.
Bridging psychological distance: The impact of immersive media on distant and proximal environmental issues
2021, Computers in Human Behavior
The growing psychological distance from environmental issues is often used to explain the lack of pro-environmental attitudes and behaviors in today's society. According to construal level theory, individuals might perceive distant issues as more abstract and personally non-relevant, compared with issues they experience directly. Consequently, this experimental study (N = 112) analyzes whether immersive media, such as omnidirectional 360° videos, may be a promising approach for decreasing individuals' psychological distance and increasing personal relevance as well as perceived risk severity and behavioral intentions concerning environmental issues. The results indicate that higher levels of technological immersiveness may be especially well suited to communicate the severity of distant risks and enhance pro-environmental behavioral intentions by simulating direct experiences; however, higher levels of immersiveness seem to be detrimental for issues that were already perceived as proximal and thus may be considered too close for comfort.

View all citing articles on Scopus

View full text

Virtual reality check: Statistical power, reported results, and the validity of research on the psychology of virtual reality and immersive environments

Highlights

Abstract

Section snippets

Declaration of competing interests

Defining virtual reality: approaches and challenges

Materials and method

Descriptive statistics

Discussion

Conclusion

Behaviour Research and Therapy

Moving from virtual reality exposure-based therapy to augmented reality exposure-based therapy: A review

Frontiers in Human Neuroscience

Communication within virtual reality: Creating a space for research

Journal of Communication

The vision of virtual reality

A power primer

Psychological Bulletin

Effect of positive findings on submission and acceptance rates: A note on meta-analysis bias

Professional Psychology: Research and Practice

How immersive is enough? A meta-analysis of the effect of immersive technology on user presence

Media Psychology

The science of technology and human behavior: Standards, old and new

Journal of Media Psychology

statcheck: Extract statistics from articles and recompute p values

Virtual reality: A survival guide for the social scientist

Journal of Media Psychology

Virtual experiences, physical behaviors: The effect of presence on imitation of an eating avatar

Presence

Misconceptions, misuses, and misinterpretations of p values and significance testing

The Journal of Bone and Joint Surgery

Why does the replication crisis seem worse in psychology? Slate

Explicating sensationalism in television news: Content and the bells and whistles of form

Journal of Broadcasting & Electronic Media

European Journal of Epidemiology

Consequences of prejudice against the null hypothesis

Psychological Bulletin

Virtual reality in mental health: A review of the literature

Social Psychiatry and Psychiatric Epidemiology

Health communication research in the digital age: A systematic review

Journal of Communication in Healthcare

Why most published research findings are false

PLoS Medicine

Measuring the prevalence of questionable research practices with incentives for truth telling

Psychological Science

The virtual doppelganger: Effects of a virtual reality simulator on perceptions of schizophrenia

The Journal of Nervous and Mental Disease

On effect size

Psychological Methods

Practical significance: A concept whose time has come

Educational and Psychological Measurement

Virtual reality therapy versus cognitive behavior therapy for social phobia: A preliminary controlled study

CyberPsychology and Behavior

The probability of p-values as a function of the statistical power of a test

An insider's view of the future of virtual reality

Journal of Communication

Immersive virtual environment technology as a basic research tool in psychology

Behavior Research Methods, Instruments, & Computers