When does social desirability become a problem? Detection and reduction of social desirability bias in information systems research

Abstract

Social desirability (SD) bias occurs in self-report surveys when subjects give socially desirable responses by over- or underreporting their behavior. Despite knowledge of SD as a potential threat to the validity of information systems (IS) research, little has been done to systematically assess its extent. Furthermore, we are uncertain of how to recover reliable estimates of the relationships between research variables contaminated by SD bias. We sought in this study to assess the extent of SD bias in causal inferences when independent and/or dependent variables are contaminated. We also evaluated whether an SD scale in conjunction with partial correlation could effectively and efficiently correct SD bias when it is found. To achieve these purposes, we designed a survey study and collected data from Amazon's Mechanical Turk in the context of mobile loafing, which refers to employees’ personal use of the mobile Internet during business hours. Using various detection methods, we found that SD bias existed in the context of mobile loafing. From the results of the variance reduction rate and a covariate technique, we found that SD bias becomes problematic when both the independent and dependent variables are susceptible to SD bias. Overall, our study contributes significantly to the IS literature by revealing the extent of SD bias and the magnitude of the possible correction for it in IS research.

Introduction

Surveys play a critical role in Information Systems (IS) research. A key assumption of self-report surveys is that respondents bring to mind appropriate information accurately and respond honestly [1]. Social desirability (SD) bias, one of the most frequently raised concerns with self-report surveys, is a tendency for subjects to give socially desirable responses by overreporting behaviors (e.g., knowledge contribution) that make them look good and underreporting behaviors (e.g., cyberbullying) that make them look bad [2], [3], [4]. This bias is problematic because it distorts the means and variances of research variables and threatens the validity of survey-based causal inferences in IS research [5,6]. For these reasons, conclusions drawn from analyses using potentially biased self-report data can misguide managerial and policy decisions [6,7]. Thus, it is crucial for researchers and practitioners to assess the extent of SD bias and, if necessary, control for it in drawing inferences from their survey data.

Recent IS studies have attempted to assess the extent of SD bias when researchers examine sensitive topics such as addiction, software piracy, unethical programming, and cyberbullying [5,6,[8], [9], [10], [11], [12], [13], [14], [15], [16], [17]–18]. Notable methods to assess and/or reduce SD bias in prior IS research are (1) indirect questioning, (2) a randomized response technique (RRT), (3) implicit constructs and measures, and (4) the SD scale and covariate techniques. First, indirect questioning is a projective technique in which subjects are asked to report the thoughts or actions of others similar to themselves [19]. Second, the RRT seeks to prevent SD bias during data collection by protecting privacy through a randomization device with a known probability distribution (e.g., coins, dice) [6,20]. Third, implicit constructs that are only indirectly measured can reduce SD bias because implicit constructs do not depend on explicit and direct measures. Fourth, researchers can include an SD scale in a survey questionnaire and then assess SD bias by examining the correlations between the SD scale and research variables [9,21]. If significant correlations are found, a covariate technique using partial correlation can be used to reduce SD bias [[22], [23]–24].

Previous IS researchers have relied on one or other of the alternative approaches to assess the extent of SD bias, but their conclusions have been contradictory. For example, some research, based on the relatively low correlations between an SD scale and sensitive variables, indicated SD bias was not a problem [8,[13], [14]–15,17]. In contrast, Kwan et al. [6] found the structural paths of respondents with direct questioning were significantly higher than those of respondents with the RRT and thus concluded that SD bias is a real threat in IS research. Similarly, in social psychology, some researchers considered SD bias significant, but others believed it to be trivial [25]. These contradictory conclusions generally hinged on the context and contingencies of admitting to undesirable behavior. For example, anger measures from parolees are significantly susceptible to SD bias [26], whereas admitting anger among college students is not [27].

To resolve these contradictions about SD bias, Paunonen and LeBel [25] used a Monte Carlo simulation to evaluate its effects in an independent variable on an unbiased dependent variable. They concluded that SD bias is not seriously problematic, at least not in a scenario in which only one variable is biased. Nevertheless, despite their findings, the extent of SD bias remains uncertain when both independent and dependent variables are susceptible to it. Although various methods to detect this bias have been proposed, prior IS research has relied heavily on correlation analysis that used a short form of the Marlowe–Crowne SD scale. Furthermore, we remain unsure how to effectively control for SD bias, especially when the RRT is not viable. It is especially important to understand how effective the technique of using SD scales could be in recovering reliable estimates of the relationships between contaminated research variables.

Our main objective in this study was twofold: First, it seeks to assess the extent of SD bias in the estimation of causal relationships when it contaminates independent and/or dependent variables. Second, it evaluates the effectiveness of SD scales with partial correlations in controlling SD bias and acquiring reliable estimates of causal relationships. To achieve these goals, we designed a survey study and collected data from Amazon's Mechanical Turk (MTurk) in the context of mobile loafing, or “nonwork-related mobile computing,” which refers to employees’ personal use of the mobile Internet during business hours [[28], [29], [30]–31]. Our survey is intended to examine how the estimation of causal relationships varies depending on the different conditions of SD bias in independent and dependent variables. This study used indirect questioning, correlation analysis, and a factor mixture model (FMM) to assess the extent of SD bias. It was also designed to help understand how useful a covariate technique is in recovering true causal relationships free of SD bias. To correct for SD bias, we used the methods of a variance reduction rate (VRR) and a covariate technique.

This study contributes to IS research in several ways. First, by venturing beyond the one-biased-variable condition examined in Paunonen and LeBel [25], we contribute to the literature by examining a two-biased-variable condition. Our contribution lies in our finding that, although the level of distortion in causal inferences is minor in a one-biased-variable condition—which is consistent with the prior literature—a two-biased-variable condition may cause a problematic degree of change in causal inferences. Second, our results can resolve the different conclusions of the effects of SD bias in previous IS research. In particular, a one-biased-variable condition supports the argument that SD bias is not problematic (e.g., [13,15]), but a two-biased-variable condition supports the conclusion that SD bias may threaten the validity of self-report research (e.g., [6]). Third, unlike prior IS research that relied on a single method to assess SD bias, our study used—and evaluated—multiple methods. Among these various methods, we found that SD scales are more useful than other techniques because they can assess the degree of SD bias and control it by using partial correlations. Finally, we found that the impression management scale is efficient and effective in measuring and controlling for SD bias in the contexts of negative use of information technology (IT) such as addictive IT use, cyberbullying, and digital piracy.

This article is organized as follows. In the next section, we review SD bias and techniques for reducing it. Then, we present an empirical model for assessing and controlling SD bias and a discussion of our methods. Next, we examine four methods to detect SD bias and a covariate technique to reduce it. Finally, we conclude with theoretical, methodological, and practical contributions, along with limitations of our study and directions for future research.