Cross-sectional research: A critical perspective, use cases, and recommendations for IS research

Highlights

• Cross-sectional study designs are predominant in Information Systems research.

• Cross-sectional research follows an efficient and inexpensive execution.

• Cross-sectional data limit statements related to temporal causality.

• Use configurational or mixed-method studies when having cross-sectional data.

• We offer four recommendations for when and how to use cross-sectional data.

Abstract

Cross-sectional data is pervasive in information systems (IS) research. This editorial reviews cross-sectional studies, summarizes their strengths and limitations, and derives use cases of when cross-sectional data is and is not useful in answering research questions. We raise concerns about assertions of temporal causality using data collected employing cross-sectional methods with no temporal order, which makes cause and effect difficult to establish. Based on our discussion of research using cross-sectional data and its limitations, we offer four recommendations for when and how to use such data: (1) improve credibility by reporting research in detail and transparently, (2) ensure appropriate sampling, (3) take configurational perspectives, and (4) integrate cross-sectional data into mixed- or multi-method designs. By doing so, we help IS researchers position and use cross-sectional studies appropriately within their methodological repertoire.

Introduction

In order to prepare a tasty dish, a chef needs a clear and good recipe, carefully selected ingredients, and a well-equipped kitchen. The dish’s flavor and presentation depend on the chef attending to all three elements of its preparation – following all the proper steps, using high quality ingredients with good flavor, and having all the tools needed to prepare the meal. If the chef skips a step, uses low-quality ingredients, or lacks an essential tool, the dish may well be a failure.

Good research is analogous to cooking. Studies are grounded in clear and solid theory, analyze meaningful data (sets), and apply research methods properly. Numerous Information systems (IS) researchers provide careful guidance on how to develop theory (e.g., contextualization; Hong et al., 2014), advice on how to gather data (Straub, 1989), and when to apply research methods (e.g., qualitative comparative analysis; Mattke et al., 2022; Mattke et al., 2021; Pappas & Woodside, 2021) that can help scholars develop compelling explanations for real-world phenomena. However, if scholars fail to pay sufficient attention to how well different aspects of the research align, e.g., with the research questions and data method, researchers risk overstating conclusions or drawing incorrect inferences.

Data sets are the least elaborated element in IS research. Scholars often fail to sufficiently explain the implications of the data collection technique on theory testing or formally test whether their data meets the quality thresholds needed to conduct analysis. As a result, scholars sometimes rely on sub-optimal data, overuse one form of data collection, or make greater inferences than the data support, an especially great danger with regard to cross-sectional data.

A casual review of research published in major IS journals reveals the pervasiveness of using cross-sectional data. In the International Journal of Information Management (IJIM), for example, more than 85 % of papers published from 2020 to 2022 employ a cross-sectional research design. Editors-in-chief of various journals across different subject disciplines have raised concerns about overusing any one form of data collection (Dwivedi, Hughes, Cheung, et al., 2022). One common mistake is using cross-sectional data inappropriately to answer research questions that implicitly or explicitly point to temporal causality (Bono & McNamara, 2011). Consider a research question exploring whether and how beliefs and intentions shape behaviors. Cross-sectional survey data cannot be analyzed to establish causal relationships, but rather only associations between concepts. Temporal causality between beliefs, intentions, and behavior and how variance in beliefs and intentions relate to behavior can be better proven using multi-wave data (Kim, 2009, Kim and Malhotra, 2005).

Since the great majority of IS research attempts to establish temporal causality (e.g., focuses on cause-effect relationships) or assesses how temporal causality evolves over time, and since most recent IS research analyzes cross-sectional data, it is valuable to review the use of cross-sectional data in IS research. In this editorial, we assert the need in IS research to employ multiple research designs, such as collecting data from the same people in two (Shirish et al., 2021) or more (Adamovic, 2022) waves or using experiments to establish causality (Fink, 2022). While cross-sectional data can be analyzed to demonstrate associations between concepts, causality and cause-effect relationships can be demonstrated effectively using research designs that allow for the temporal ordering of variables.

This editorial provides an overview of cross-sectional studies, presents the strengths and weaknesses of using such data, and provides recommendations for conducting cross-sectional research, thus contributing to IS theory and practice. We hope that this critical review of the use of cross-sectional data in IS research will stimulate a broader conversation about data collection practices.