The language-as-fixed-effect fallacy: A critique of language statistics in psychological research

doi:10.1016/S0022-5371(73)80014-3

Journal of Verbal Learning and Verbal Behavior

Volume 12, Issue 4, August 1973, Pages 335-359

https://doi.org/10.1016/S0022-5371(73)80014-3 Get rights and content

Current investigators of words, sentences, and other language materials almost never provide statistical evidence that their findings generalize beyond the specific sample of language materials they have chosen. Nevertheless, these same investigators do not hesitate to conclude that their findings are true for language in general. In so doing, it is argued, they are committing the language-as-fixed-effect fallacy, which can lead to serious error. The problem is illustrated for one well-known series of studies in semantic memory. With the appropriate statistics these studies are shown to provide no reliable evidence for most of the main conclusions drawn from them. A review of other experiments in semantic memory shows that many of them are likewise suspect. It is demonstrated how this fallacy can be avoided by doing the right statistics, selecting the appropriate design, and sampling by systematic procedures, or, alternatively, by proceeding according to the so-called method of single cases.

References (64)

BransfordJ.D. et al.
Sentence memory: A constructive versus interpretive approach
Cognitive Psychology
(1972)
ClarkE.V.
On the acquisition of the meaning of before and after
Journal of Verbal Learning and Verbal Behavior
(1971)
ClarkH.H. et al.
On the process of comparing sentences against pictures
Cognitive Psychology
(1972)
CollinsA.M. et al.
Retrieval time from semantic memory
Journal of Verbal Learning and Verbal Behavior
(1969)
CollinsA.M. et al.
Does category size affect categorization time?
Journal of Verbal Learning and Verbal Behavior
(1970)
CollinsA.M. et al.
Facilitating retrieval from semantic memory: The effect of repeating part of an inference
Acta Psychologica
(1970)
FreedmanJ.L. et al.
Retrieval of words from long-term memory
Journal of Verbal Learning and Verbal Behavior
(1971)
LandauerT.K. et al.
Information-retrieval from long-term memory: Category size and recognition time
Journal of Verbal Learning and Verbal Behavior
(1968)
LandauerT.K. et al.
Category size and semantic-memory retrieval
Journal of Verbal Learning and Verbal Behavior
(1972)
LoftusE.F. et al.
Effect of category-name frequency on the speed of naming an instance of the category
Journal of Verbal Learning and Verbal Behavior
(1972)

RipsL.J. et al.

Semantic distance and the verification of semantic relations

Journal of Verbal Learning and Verbal Behavior

(1973)

RubensteinH. et al.

Homographic entries in the internal lexicon

Journal of Verbal Learning and Verbal Behavior

(1970)

RubensteinH. et al.

Homographic entries in the internal lexicon: Effects of systematicity and relative frequency of meanings

Journal of Verbal Learning and Verbal Behavior

(1971)

RubensteinH. et al.

Evidence for phonemic recoding in visual word recognition

Journal of Verbal Learning and Verbal Behavior

(1971)

SmithE.E. et al.

Retrieval of artificial facts from long-term memory

Journal of Verbal Learning and Verbal Behavior

(1972)

WilkinsA.

Conjoint frequency, category size, and categorization time

Journal of Verbal Learning and Verbal Behavior

(1971)

AndersonS.R.

How to get even

Language

(1972)

BattigW.F. et al.

Category norms for verbal items in 56 categories: A replication and extension of the Connecticut category norms

Journal of Experimental Psychology Monograph

(1969)

BendixE.H.

Componential analysis of general vocabulary: The semantic structure of a set of verbs in English, Hindi, and Japanese

(1966)

BierwischM.

Some semantic universals of German adjectivals

Foundations of Language

(1967)

CarrollJ.B.

An experiment in evaluating the quality of translations

Mechanical Translation and Computational Linguistics

(1966)

ChomskyC.

The acquisition of syntax in children from 5 to 10

(1969)

Clark, E. V. Normal states and evaluative view-points: More on come and go. Language, in...

ClarkH.H.

Linguistic processes in deductive reasoning

Psychological Review

(1969)

ClarkH.H. et al.

The use of syntax in understanding sentences

British Journal of Psychology

(1968)

ClarkH.H. et al.

The role of semantics in remembering comparative sentences

Journal of Experimental Psychology

(1969)

ClarkH.H. et al.

Semantic distinctions and memory for complex sentences

Quarterly Journal of Experimental Psychology

(1968)

ColemanE.B.

Generalizing to a language population

Psychological Reports

(1964)

ColemanE.B.

Learning of prose written in four grammatical transformations

Journal of Applied Psychology

(1965)

CollinsA.M. et al.

Categories and subcategories in semantic memory

ConradC.

Cognitive economy in semantic memory

Journal of Experimental Psychology

(1972)

DeSotoC. et al.

Social reasoning and spatial paralogic

Journal of Personality and Social Psychology

(1965)

Cited by (1677)

Crosslinguistic evidence against interference from extra-sentential distractors
2024, Journal of Memory and Language
Cue-based retrieval theories of sentence processing posit that long-distance dependency formation is guided by a cue-based retrieval mechanism: dependents are retrieved via retrieval cues associated with a verb. When retrieval cues match multiple similar items in memory, this leads to cue-based retrieval interference. A landmark study by Van Dyke and McElree tested interference from sentence-external items: retrieval cues were manipulated to (mis-)match semantically similar items presented prior to a target dependency. The support for interference of this type is weak, and only comes from English object cleft constructions. Our study provides a cross-linguistic investigation of interference from sentence-external items: Three eyetracking studies in English, German and Russian tested interference in the online processing of filler-gap dependencies under varying task demands. A fourth study attempted to replicate the Van Dyke and McElree study using self-paced reading. Bayes factors analyses show cross-linguistic evidence against interference from sentence-external items. A broader implication from these data is that cue-based retrieval interference is driven by sentence-internal distracting items, suggesting that a cue-based search is restricted to the current linguistic context.
The effects of emotional stimuli on Word retrieval in people with aphasia
2024, Neuropsychologia
Prior studies have shown that people with aphasia (PWA) have demonstrated superior language performance for emotional compared to nonemotional stimuli on a range of tasks, including auditory comprehension, verbal pragmatics, repetition, reading, and writing. However, studies on word retrieval, specifically, have suggested a possible interference effect of emotion on naming. The purpose of this study was to examine the effect of the emotional valence of stimuli on word retrieval in a series of naming tasks in PWA.
Thirteen PWA and 13 neurotypical controls participated in four single-word naming tasks, including 1) object picture naming, 2) action picture naming, 3) category-member generation, and 4) verb generation. Each task included three valence sets of positively-, negatively-, and neutrally-rated pictures or words, which were obtained from the standardized International Affective Picture System (Lang et al., 2008) and the Affective Norms for Emotional Words (Bradley and Lang, 1999) databases. Accuracy and reaction time (RT) were measured and compared across groups, tasks, and valence sets.
Emotional stimuli, especially negative stimuli, resulted in worse naming performance, as measured by accuracy and RT, compared to nonemotional stimuli in PWA and neurotypical controls. This effect was relatively robust across the four naming tasks. In most cases, negative stimuli resulted in lower accuracy and slower RT than positive stimuli.
These findings suggest that stimulus valence may interfere with word retrieval for PWA and neurotypical adults and that this effect is robust across different types of naming tasks that vary by word class (nouns versus verbs) and stimulus type (pictures versus words). Negative stimuli resulted in worse naming performance than positive stimuli. These results suggest that emotionality of stimuli is an important variable to consider in word retrieval research.
lmeEEG: Mass linear mixed-effects modeling of EEG data with crossed random effects
2024, Journal of Neuroscience Methods
Mixed-effects models are the current standard for the analysis of behavioral studies in psycholinguistics and related fields, given their ability to simultaneously model crossed random effects for subjects and items. However, they are hardly applied in neuroimaging and psychophysiology, where the use of mass univariate analyses in combination with permutation testing would be too computationally demanding to be practicable with mixed models.
Here, we propose and validate an analytical strategy that enables the use of linear mixed models (LMM) with crossed random intercepts in mass univariate analyses of EEG data (lmeEEG). It avoids the unfeasible computational costs that would arise from massive permutation testing with LMM using a simple solution: removing random-effects contributions from EEG data and performing mass univariate linear analysis and permutations on the obtained marginal EEG.
lmeEEG showed excellent performance properties in terms of power and false positive rate.
lmeEEG overcomes the computational costs of standard available approaches (our method was indeed more than 300 times faster).
lmeEEG allows researchers to use mixed models with EEG mass univariate analyses. Thanks to the possibility offered by the method described here, we anticipate that LMM will become increasingly important in neuroscience. Data and codes are available at osf.io/kw87a. The codes and a tutorial are also available at github.com/antovis86/lmeEEG.
Modeling the bilingual advantage: Do results differ between analysis methods?
2023, Ampersand
Whether bilingualism results in improved executive function is controversial. According to some researchers, putative bilingual advantages can be explained by individual differences in unmeasured non-linguistic variables. Additionally, commonly used models containing exclusively fixed-effects do not account for the data structure inherent in multi-trial behavioral tasks. Mixed-effects models by contrast address both of these issues, allowing for a more valid test of the bilingual advantage hypothesis. The present study aimed to assess whether the choice of analysis method impacted on results when investigating the bilingual advantage in executive function. Simon task data from Mandarin-English speaking Chinese adolescents were analyzed using separate fixed and mixed-effects models. Comparisons between models revealed considerable differences in the pattern of results. Most notable was the observation that a number of previously significant main effects on task performance were no longer significant in a mixed model accounting for the inclusion of multiple trials for each participant. Implications for the bilingual advantage are discussed.
How to present L2 Chinese words effectively for learning: Exploring learning outcomes and learner perceptions
2024, Studies in Second Language Acquisition
Don't let perfect be the enemy of better: In defense of unparameterized megastudies
2024, Behavioral and Brain Sciences

View all citing articles on Scopus

¹: The preparation of this paper was supported in part by Public Health Service Grant MH-20021 from the National Institute of Mental Health. I am very grateful to William P. Banks, J. Merrill Carlsmith, Eve V. Clark, Douglas J. Herrmann, Peter Lucy, Lance J. Rips, and Edward J. Shoben for their helpful comments on the manuscript and to Thomas K. Landauer, David E. Meyer, and three anonymous reviewers for their detailed reviews of the paper. I am especially indebted to Edward E. Smith and Ewart A. C. Thomas for their generous and thoughtful counsel on many points in the paper.

View full text

The language-as-fixed-effect fallacy: A critique of language statistics in psychological research

Cognitive Psychology

Journal of Verbal Learning and Verbal Behavior

Cognitive Psychology

Journal of Verbal Learning and Verbal Behavior

Journal of Verbal Learning and Verbal Behavior

Acta Psychologica

Journal of Verbal Learning and Verbal Behavior

Journal of Verbal Learning and Verbal Behavior

Journal of Verbal Learning and Verbal Behavior

Journal of Verbal Learning and Verbal Behavior

Journal of Verbal Learning and Verbal Behavior

Journal of Verbal Learning and Verbal Behavior

Journal of Verbal Learning and Verbal Behavior

Journal of Verbal Learning and Verbal Behavior

Journal of Verbal Learning and Verbal Behavior

Journal of Verbal Learning and Verbal Behavior

How to get even

Language

Category norms for verbal items in 56 categories: A replication and extension of the Connecticut category norms

Journal of Experimental Psychology Monograph

Componential analysis of general vocabulary: The semantic structure of a set of verbs in English, Hindi, and Japanese

Some semantic universals of German adjectivals

Foundations of Language

An experiment in evaluating the quality of translations

Mechanical Translation and Computational Linguistics

The acquisition of syntax in children from 5 to 10

Linguistic processes in deductive reasoning

Psychological Review

The use of syntax in understanding sentences

British Journal of Psychology

The role of semantics in remembering comparative sentences

Journal of Experimental Psychology

Semantic distinctions and memory for complex sentences

Quarterly Journal of Experimental Psychology

Generalizing to a language population

Psychological Reports

Learning of prose written in four grammatical transformations

Journal of Applied Psychology

Categories and subcategories in semantic memory

Cognitive economy in semantic memory

Journal of Experimental Psychology

Social reasoning and spatial paralogic

Journal of Personality and Social Psychology