Why Are Self-Report and Behavioral Measures Weakly Correlated?

doi:10.1016/j.tics.2020.01.007

Trends in Cognitive Sciences

Volume 24, Issue 4, April 2020, Pages 267-269

https://doi.org/10.1016/j.tics.2020.01.007 Get rights and content

Accumulating evidence indicates weak correlations between self-report and behavioral measures of the same construct. We suggest that these weak correlations result from the poor reliability of many behavioral measures and the distinct response processes involved in the two measurement types. We also describe how researchers can benefit from appropriate use of these measures.

Introduction

Self-report and behavioral measures are two of the most popular methods of capturing individual differences in psychology. The same psychological construct is often assessed with both types of measures, with researchers using them interchangeably, often conflating findings across measurement type. However, across a series of domains, recent meta-analyses and large-scale investigations have consistently found that self-report and behavioral measures of the same construct were weakly correlated. For example, the average correlation between self-report and behavioral measures of self-control [1,2], emotional intelligence [3], empathy [4], risk preference [5], and creativity [6] ranged from 0 to 0.20, indicating a weak (or nonexistent) association between these two types of measures of the presumed ‘same’ construct.

This weak association suggests that self-report and behavioral measures might be inherently different and thus cannot be considered interchangeable indicators of a single construct. Our goal here is to: (i) explain why self-report and behavioral measures are bound to be weakly correlated by paying careful attention to the psychometric properties of these measures; and (ii) describe how researchers can benefit from appropriate use of them in research.

Section snippets

The Reasons for Weak Correlations

The reasons for weak correlations are both methodological and conceptual.

Appropriate Applications in Research

Understanding these methodological and conceptual differences could lead to a better understanding of how to use these measures in research. Measures with high reliability (i.e., most self-report measures and select behavioral measures such as the working memory span task) can be used to predict individual differences in real-life outcomes, and reliable self-report and behavioral measures may explain incremental variance above each other because they are likely to assess different constructs.

Concluding Remarks

The weak correlations between self-report and behavioral measures of the presumed same construct result from the poor reliability of many behavioral measures and the distinct response processes involved in these two measurement types. We suggest that only measures with high reliability be used for individual difference research, whereas measure with low reliability may help to predict the short-term waxing and waning of an attribute for the same individual.

Acknowledgments

J.D. was supported by the Swedish Research Council (2018-06664). K.M.K. was supported by a grant from the National Institute on Drug Abuse (DA047247). M.I. was supported by grants from the Social Sciences and Humanities Research Council of Canada (#435-2019-0144) and from Canada’s Natural Sciences and Engineering Research Council (RGPIN-2019-05280).

References (14)

P.J. Kajonius et al.
Individuals with dark traits have the ability but not the disposition to empathize
Pers. Individ. Differ.
(2020)
S. Noble
A decade of test–retest reliability of functional connectivity: a systematic review and meta-analysis
Neuroimage
(2019)
A.H. Egbert
Executive function and dietary intake in youth: a systematic review of the literature
Appetite
(2019)
B. Saunders
Reported self-control is not meaningfully associated with inhibition-related executive function: a Bayesian analysis
Collabra Psychol.
(2018)
I.W. Eisenberg
Uncovering the structure of self-regulation through data-driven ontology discovery
Nat. Commun.
(2019)
D.L. Joseph et al.
Emotional intelligence: an integrative meta-analysis and cascading model
J. Appl. Psychol.
(2010)
B.A. Murphy et al.
Are self-report cognitive empathy ratings valid proxies for cognitive empathy ability? Negligible meta-analytic relations with behavioral task performance
Psychol. Assess.
(2019)

There are more references available in the full text version of this article.

Cited by (290)

Attentional Biases and Their Association with Substance-Use-Related Problems and Addictive Behaviors: The Utility of a Gamified Value-Modulated Attentional Capture Task
2024, Addictive Behaviors Reports
Attentional biases towards reward stimuli have been implicated in substance use-related problems. The value-modulated attentional capture (VMAC) task assesses such reward-related biases. The VMAC task widely used in lab studies tends to be monotonous and susceptible to low effort. We therefore tested a gamified online version of the VMAC that aimed to increase participant engagement. Our goal was to examine how VMAC is associated with substance use-related problems and addictive behaviors, and whether this association is moderated by cognitive control.
We recruited 285 participants from an online community, including heavy alcohol users. All participants completed a novel gamified version of the VMAC task, measures of substance use and addictive behaviors (addictive-like eating behavior, problematic smartphone use), the WebExec measure of problems with executive functions, and the Stroop Adaptive Deadline Task (SDL) as a measure of cognitive control.
The gamified VMAC task successfully identified value-modulated attentional capture effects towards high-reward stimuli. We found no significant associations between VMAC scores, problematic alcohol or cannabis use, addictive behaviors, or any moderation by a behavioral measure of cognitive control. Exploratory analyses revealed that self-reported cognitive problems were associated with more alcohol-, and cannabis-related problems, and addictive behaviors. Greater attentional capture (VMAC) was associated with more cannabis use-related problems among individuals with higher levels of self-reported cognitive problems.
Our study is one of the first to demonstrate the utility of the gamified version of the VMAC task in capturing attentional reward biases. Self-reported problems with cognitive functions represent a key dimension associated with substance use-related problems and addictive behaviors.
Prefrontal activity during IOWA Gambling Task in young adult women
2024, Behavioural Brain Research
This study aims to investigate the relationships between personality traits of impulsivity, using the UPPS-P Impulsive Behaviour Scales shortened version, and prefrontal cortex (PFC) activity during the IOWA Gambling Task (IGT) in young adult women. The study included a sample of 83 young, healthy females (19.8 ± 1.4 years), who voluntarily took part in the study. Repeated measures analysis during the IGT revealed a significant increase in HbO (all p <.001; η_p² >.31) and a decrease in Hbr (all p <.003; η_p² >.08) in all prefrontal quadrants. This increase in oxygenation occurs primarily during the choice period under ambiguity (r =.23; p =.039). Additionally, there was a significant linear decrease in selecting the decks associated with a high frequency of losses (p <.001), while the favorable deck with low losses showed a linear increase (F = 12.96; p <.001). Notably, discrepancies were found between UPPS-P and IGT impulsivity ratings. The Lack of Perseverance and Lack of Premeditation scales from the UPPS-P were identified as significant predictors of HbO levels, mainly in the two quadrants of the left hemisphere’s, lateral (adjusted R² =.23; p <.001; f² =.34) and rostral (adjusted R² =.13; p <.002; f² =.17). These findings suggest that young adult women predominantly adopt a punishment^-avoidance strategy during IGT, exhibiting increased activation in the left hemisphere, especially during the task’s initial phase characterized by ambiguity.
Moral violations that target more valued victims elicit more anger, but not necessarily more disgust
2024, Journal of Experimental Social Psychology
The same moral violation can give rise to different emotional and behavioral responses in different individuals. The mechanisms that give rise to such differences – and the functions that those mechanisms serve – are unclear. Previous work suggests that people experience greater anger toward violations that target themselves or kin than those that target others, whereas they experience greater disgust toward violations that target others than those that target themselves or kin. In turn, anger has a stronger relation with direct aggression than indirect aggression, and disgust a stronger relation with indirect aggression than direct aggression. The current study tests whether these patterns depend on the value observers place on the targets of moral violations, even within folk relationship categories. In two studies, we asked participants to think of a person they know and to imagine that person being targeted by a moral violation described in a vignette. We assessed the value that participants placed on the target using a financial tradeoff task, their emotional reaction to the violation, and their desires to aggress toward the perpetrator. Results revealed that: (1) interpersonal value relates more strongly to anger than disgust toward the moral violation; (2) interpersonal value relates more strongly to direct than indirect aggression motives; and (3) anger relates to both direct and indirect aggression motives, whereas disgust relates only to indirect aggression motives. These results suggest that the value one places on the victims of moral violations influences emotional and behavioral reactions to those violations.
Unpacking the overlap between Autism and ADHD in adults: A multi-method approach
2024, Cortex
The overlap between Autism and Attention-Deficit Hyperactivity Disorder (ADHD) is widely observed in clinical settings, with growing interest in their co-occurrence in neurodiversity research. Until relatively recently, however, concurrent diagnoses of Autism and ADHD were not possible. This has limited the scope for large-scale research on their cross-condition associations, further stymied by a dearth of open science practices in the neurodiversity field. Additionally, almost all previous research linking Autism and ADHD has focused on children and adolescents, despite them being lifelong conditions. Tackling these limitations in previous research, 5504 adults – including a nationally representative sample of the UK (Study 1; n = 504) and a large pre-registered study (Study 2; n = 5000) – completed well-established self-report measures of Autism and ADHD traits. A series of network analyses unpacked the associations between Autism and ADHD at the individual trait level. Low inter-item connectivity was consistently found between conditions, supporting the distinction between Autism and ADHD as separable constructs. Subjective social enjoyment and hyperactivity-impulsivity traits were most condition-specific to Autism and ADHD, respectively. Traits related to attention control showed the greatest Bridge Expected Influence across conditions, revealing a potential transdiagnostic process underlying the overlap between Autism and ADHD. To investigate this further at the cognitive level, participants completed a large, well-powered, and pre-registered study measuring the relative contributions of Autism and ADHD traits to attention control (Study 3; n = 500). We detected age- and sex-related effects, however, attention control did not account for the covariance between Autism and ADHD traits. We situate our findings and discuss future directions in the cognitive science of Autism, ADHD, and neurodiversity, noting how our open datasets may be used in future research.
The effectiveness of Go/No-Go and Stop-Signal training in reducing food consumption and choice: A systematic review and meta-analysis
2024, Appetite
The Go/No-Go and Stop-Signal tasks have been used to reduce excess food intake via repeated pairing of food cues with response inhibition. A meta analysis of 32 studies was conducted to determine whether, and under which conditions, the Go/No-Go and Stop-Signal training tasks are effective in reducing food consumption or choice. Moderators included task parameters (e.g., number of sessions, stop signal), sample differences (e.g., age, weight), and the measure of food consumption or choice. Overall, there was a small effect for Go/No-Go and Stop-Signal training in reducing food consumption or choice, g = −0.21, CI₉₅ = [-0.31, −0.11], p < .001, with this holding individually only for a single session of the Go/No-Go Task, g = −0.31, CI₉₅ = [-0.45, −0.18], p < .001. Comprehensive investigation of the impact of varying moderators indicated that the effect for Go/No-Go training was robust. Nevertheless, there was significant variation in the specific parameters of the task. Overall, the present meta-analysis extends previous findings by providing comprehensive evidence that the Go/No-Go Task is effective in reducing food consumption and choice, as well as providing optimal parameter recommendations for the task.
Developing the Japanese and English versions of the Mind Blanking Questionnaire (MBQ): validation and reliability
2024, Personality and Individual Differences
We occasionally experience moments when our minds go blank, or our attention disappears. This psychological phenomenon, known as mind blanking (MB), has recently garnered increased attention. MB has been assessed at the state level using experience sampling methods in which participants undertake a cognitive task and probes suddenly appear asking them to report the current contents of their thoughts. However, it may be possible to evaluate MB at the “trait level” as is the case with other mental states (e.g., mind wandering, anxiety, and the like). In the present study, we developed a new scale, the Mind Blanking Questionnaire (MBQ), for assessing the tendency of MB at the trait level in both Japanese and English. The MBQ exhibited good psychometric properties including internal consistency, test-retest reliability, and criterion and construct validities. Additionally, it displayed measurement invariance between the language versions, genders, and age groups. The MBQ would be a valuable tool to assess the individual tendency of MB and contribute to cross-cultural studies.

View all citing articles on Scopus

View full text

Trends in Cognitive Sciences

ForumWhy Are Self-Report and Behavioral Measures Weakly Correlated?

Introduction

Section snippets

The Reasons for Weak Correlations

Appropriate Applications in Research

Concluding Remarks

Acknowledgments

Pers. Individ. Differ.

Neuroimage

Appetite

Reported self-control is not meaningfully associated with inhibition-related executive function: a Bayesian analysis

Collabra Psychol.

Uncovering the structure of self-regulation through data-driven ontology discovery

Nat. Commun.

Emotional intelligence: an integrative meta-analysis and cascading model

J. Appl. Psychol.

Are self-report cognitive empathy ratings valid proxies for cognitive empathy ability? Negligible meta-analytic relations with behavioral task performance

Psychol. Assess.

Forum
Why Are Self-Report and Behavioral Measures Weakly Correlated?