Elsevier

Brain and Language

Volume 120, Issue 2, February 2012, Pages 163-173
Brain and Language

Syntactic structure building in the anterior temporal lobe during natural story listening

https://doi.org/10.1016/j.bandl.2010.04.002Get rights and content

Abstract

The neural basis of syntax is a matter of substantial debate. In particular, the inferior frontal gyrus (IFG), or Broca’s area, has been prominently linked to syntactic processing, but the anterior temporal lobe has been reported to be activated instead of IFG when manipulating the presence of syntactic structure. These findings are difficult to reconcile because they rely on different laboratory tasks which tap into distinct computations, and may only indirectly relate to natural sentence processing. Here we assessed neural correlates of syntactic structure building in natural language comprehension, free from artificial task demands. Subjects passively listened to Alice in Wonderland during functional magnetic resonance imaging and we correlated brain activity with a word-by-word measure of the amount syntactic structure analyzed. Syntactic structure building correlated with activity in the left anterior temporal lobe, but there was no evidence for a correlation between syntactic structure building and activity in inferior frontal areas. Our results suggest that the anterior temporal lobe computes syntactic structure under natural conditions.

Introduction

The combination of words into an infinite number of complex phrases is a fundamental property of human language. Yet, despite 130 years of neuroscience research of language, the brain basis of this ability remains hotly debated. Syntactic computation, i.e. the capacity to build sentence structure, has famously been associated with the left inferior frontal gyrus (IFG) (Ben-Shachar et al., 2003, Caplan et al., 2008, Caplan et al., 2002, Dapretto and Bookheimer, 1999, Embick et al., 2000, Grodzinsky, 2001, Just et al., 1996, Stromswold et al., 1996). However, a growing body of evidence also suggests a role for the left anterior temporal lobe (aTL) in basic syntactic computation (Dronkers et al., 2004, Friederici et al., 2000, Humphries et al., 2006, Mazoyer et al., 1993, Rogalsky and Hickok, 2008, Stowe et al., 1998, Vandenberghe et al., 2002). Reconciling these divergent findings is vital for understanding the neurobiological mechanisms underlying language, as well as for developing clinical treatments of language impairment.

Syntactic processing as a whole divides into many sub-computations, including the combinatory operations that build larger phrases from smaller ones, as well as the various computations that serve to establish long-distance dependencies. In this work we focused on the localization of the basic combinatory operation, i.e., the process by which words are combined to form larger phrases, or “Merge” as it is often called in theoretical linguistics (Chomsky, 1995). Combining words into phrases is pervasive during every-day language comprehension, and we sought to measure brain activity associated with this computation while minimizing the influence of artificial experimental factors by having subjects perform a relatively every-day activity: passively listen to a story.

Traditional models of the brain basis of syntax, derived from deficit-lesion studies, have long associated the IFG with the processing of certain types of syntactically complex sentences (Zurif, 1995), yet the specific linguistic functions performed by the IFG remain a matter of significant debate (see e.g. Caplan et al., 2007, Grodzinsky, 2001 and associated commentaries). A series of neuroimaging studies has built on the deficit/lesion literature by manipulating the syntactic complexity of sentences that have a similar semantic interpretation. For example, Stromswold et al. (1996) used positron emission tomography (PET) to compare the processing of simple right-branching bi-clausal sentences (the child spilled the juice that __ stained the rug) with more complex center-embedded sentences (the juice that the child spilled __ stained the rug) and reported increased activity in Broca’s area for complex sentences. Recent research extending this work has found that Broca’s area is consistently activated by various kinds of sentences involving long-distance dependencies (Ben-Shachar et al., 2004, Ben-Shachar et al., 2003). These manipulations are confounded by different demands on working memory (Chen et al., 2006, Stowe et al., 1998), but there is also evidence that when working memory is independently manipulated, parts of Broca’s area appear to be sensitive to language-specific, but not non-linguistic, working memory demands (Santi and Grodzinsky, 2007a, Santi and Grodzinsky, 2007b). These studies suggest an important role for Broca’s area in syntactic processing of complex sentences, but they do not provide straightforward evidence for its involvement in the combinatory process that builds syntactic structure, henceforth “syntactic structure building.”

Studies that manipulate the attention of the participant, whether by varying the experimental task or the stimuli, have also been used to support a link between Broca’s area and aspects of syntactic processing. Embick, Marantz, Miyashita, O’Neil, and Sakai (2000), for example, had subjects read sentences with either grammar or spelling mistakes and asked them to indicate how many errors they observed. In another study, Dapretto and Bookheimer (1999) asked participants to make semantic same/different judgments on sentence pairs which differed in surface characteristics that were either “syntactic” (e.g. active versus passive) or “semantic” (e.g. synonyms such as lawyer and attorney). Both of these studies reported increased activity in Broca’s area for the “syntactic” task. However, it is unclear to what extent these meta-linguistic attention manipulations tap into the computations that are engaged during natural syntactic processing, limiting the ability of these results to definitively answer questions about syntactic structure building. In sum, while there is a range of evidence linking the processing of syntactically complex structures with Broca’s area, it remains open whether this part of the cortex participates in syntactic structure building.

Several studies suggest that aTL is activated instead of IFG when syntactic structure is manipulated. These studies have primarily employed a sentence versus word list protocol in which subjects were shown either word lists that lacked a syntactic structure or coherent syntactically well-formed sentences constructed from the same or similar words as in the lists (Mazoyer et al., 1993). These conditions are likely to require an equivalent degree of lexical processing, but only the sentences are assumed to engender processing associated with composing words together. In this manipulation, sentences have systematically elicited more aTL activity than word lists, a finding that has been replicated with both visual (Stowe et al., 1998, Vandenberghe et al., 2002) and auditory stimuli (Friederici et al., 2000, Humphries et al., 2006, Rogalsky and Hickok, 2008). These studies provide evidence linking the anterior temporal lobe with sentence processing, leading to the hypothesis that the aTL is involved in syntactic structure building (Friederici and von Cramon, 2001, Grodzinsky and Friederici, 2006). Sentences and word lists, however, differ in many ways that are unrelated to the presence or absence of syntactic structure, making it difficult to definitively conclude from these studies that the aTL is involved in building syntactic structure. Furthermore, reconciling these findings with those that implicate Broca’s area is particularly difficult because each literature relies on different experimental tasks.

We sought to investigate the localization of syntactic structure building under relatively naturalistic conditions, without the potential confound of artificial experimental task demands. We examined brain activity while participants listened to a story, adapting and developing a technique used to study visual processing during naturalistic viewing conditions. This approach has previously been applied while subjects watch a popular movie (Bartels & Zeki, 2004; Hasson et al., 2009, Hasson et al., 2004, Mukamel et al., 2005, Nir et al., 2007, Wilson et al., 2008). While some of these naturalistic studies have investigated what cortical regions are relevant for language processing (Bartels and Zeki, 2004, Wilson et al., 2008), they have not aimed to disentangle the different operations that contribute to comprehension.

To investigate syntactic structure building under naturalistic conditions, we had subjects listen to a story, Alice in Wonderland, while brain activity was recorded using functional magnetic resonance imaging (fMRI). We used a word-by-word measure of syntactic structure to estimate the number of structure building operations computed at each word. This was contrasted with a measure that tracked the difficulty of accessing each word from the mental lexicon. These measures were correlated with hemodynamic activity to assess how structure building and lexical access affect brain activity while listening to a story.

We estimated word-by-word syntactic structure building by counting the number of syntactic nodes used to integrate each word into the phrase structure (Frazier, 1985, Hawkins, 1994, Miller and Chomsky, 1963). This metric was chosen because it transparently relates to the process of syntactic structure building that we were targeting. Our metric resembles Yngve (1960) hypothesis that associates the depth of syntactic embedding with processing load. Unlike Yngve’s hypothesis, however, which focused on the processing demands made by holding unfinished constituents in working memory, our focus was on processing associated with structure building. Thus, our metric is designed to track the amount of structure that has to be postulated at each word. Our approach builds on Frazier (1985) who argues that “the ‘work’ involved in the syntactic processing of unambiguous sentences is to be identified with the postulation of non-terminal nodes” (p. 168).

A number of theorists have linked syntactic node count with syntactic complexity, or the difficulty of determining the syntactic structure for a particular string (Frazier, 1985, Hawkins, 1994, Miller and Chomsky, 1963). Hawkins (1994, ch. 4) reviews a large body of cross-linguistic corpus-based evidence showing a systematic preference for constructions which minimize the number of syntactic nodes per constituent, independent of the number of words. In addition, the number of syntactic nodes correlates with processing difficulty associated with relative clauses built on different types of noun phrases (see Hale, 2006: 663–664).

Syntactic node count, however, is only one of many factors that have been implicated in parsing difficulty. Other factors include effort associated with holding and accessing representations in working memory (e.g., Gibson, 1998, Gibson, 2000, Vasishth and Lewis, 2006) and the relative likelihood of applying a syntactic rule given the context (Hale, 2006, Levy, 2008). There is significant disagreement as to the relative importance of these factors in behavioral measurements of parsing difficulty, and we remain uncommitted as to the identity and weighting of these factors. Our interest is not in assessing brain activity associated with processing difficulty. Rather, this study is focused on localizing the neural basis of a specific computation: syntactic structure building. We do not assume that the amount of syntactic structure in a natural context should necessarily affect processing difficulty as measured by behavioral methods. We do, however, expect that more applications of the structure building computation should be associated with increased neurovascular activity in a region associated with this computation.

Effort associated with lexical access was estimated using word frequency. This variable robustly modulates the ease of lexical access (Balota, Cortese, Sergent-Marshall, Spieler, & Yap, 2004) and has been previously studied with neuroimaging methods (Carreiras et al., 2006, Fiebach et al., 2002, Fiez et al., 1999, Kuo et al., 2003, Prabhakaran et al., 2006).

Section snippets

Participants

Nine healthy subjects (3 women), ages 22–34, participated in the experiment. All subjects were right-handed fluent English speakers with normal audition, and all provided written informed consent. Procedures complied with the safety guidelines for MRI research and were approved by the University Committee on Activities Involving Human Subjects at New York University.

Materials

Subjects listened to a 30-min portion of the story Alice’s Adventures in Wonderland by Lewis Carroll (recording of the book on

Comprehension questionnaire

Subjects showed a high degree of accuracy in their performance on the multiple-choice comprehension questionnaire administered after the story run (M = 86.4%, SD = 12.2%). This performance can be better understood in comparison to nine control subjects who were administered the questionnaire without having participated in the story-listening experiment. Control participants performed poorly on the questionnaire (M = 44.9%, SD = 11.9%), t(16) = −7.28, p < .001). The high degree of accuracy from subjects who

Discussion

In this study we employed a novel approach using fMRI to examine the neural correlates of language processing with naturalistic stimuli. We used this method to address a controversy about the brain basis of basic syntactic computation by aiming to distinguish activity associated with word-by-word syntactic structure building.

Syntactic structure building was strongly associated with activity in the left aTL, including portions of the STG and MTG. As reviewed above, previous research which has

Conclusion

In summary, we examined the neural basis of natural syntactic and lexical processing with a novel approach which correlates the time course of brain activity with the changing linguistic properties of a naturalistic speech stimulus (a story). This method allowed us to examine the neural basis of sentence processing while minimizing the potential effects of artificial task demands. Our primary finding is that a measure of syntactic structure building is correlated with activity in the left

Acknowledgments

This work was supported by the Weizmann-NYU Demonstration Fund in Neuroscience, a Fulbright Doctoral Student Fellowships (Y.N.), the National Science Foundation Grant BCS-0545186 (L.P.), and the National Institute of Health Grant R01-MH69880 (D.J.H.). We thank Asaf Bachrach, Alec Marantz and David Poeppel for helpful discussion and feedback and Gijs Brouwer and Ilan Dinstein for assistance with data collection.

References (71)

  • A.D. Friederici et al.

    Auditory language comprehension: An event-related fMRI study on the processing of syntactic and lexical information

    Brain and Language

    (2000)
  • C.R. Genovese et al.

    Thresholding of statistical maps in functional neuroimaging using the false discovery rate

    NeuroImage

    (2002)
  • E. Gibson

    Linguistic complexity: Locality of syntactic dependencies

    Cognition

    (1998)
  • Y. Grodzinsky et al.

    Neuroimaging of syntax and syntactic processing

    Current Opinion in Neurobiology

    (2006)
  • M. Gurjanov et al.

    Grammatical priming of inflected nouns by inflected adjectives

    Cognition

    (1985)
  • W.-J. Kuo et al.

    Frequency effects of Chinese character processing in the brain: An event-related fMRI study

    NeuroImage

    (2003)
  • R. Levy

    Expectation-based syntactic comprehension

    Cognition

    (2008)
  • Y. Nir et al.

    Coupling between neuronal firing rate, gamma LFP, and BOLD fMRI is related to interneuronal correlations

    Current Biology

    (2007)
  • R. Prabhakaran et al.

    An event-related fMRI investigation of phonological-lexical competition

    Neuropsychologia

    (2006)
  • L. Pylkkänen et al.

    Tracking the time course of word recognition in MEG

    Trends in Cognitive Science

    (2003)
  • J. Rademacher et al.

    Probabilistic mapping and volume measurement of human primary auditory cortex

    NeuroImage

    (2001)
  • A. Santi et al.

    Working memory and syntax interact in Broca’s area

    NeuroImage

    (2007)
  • S.J. Segalowitz et al.

    Lexical access of function versus content words

    Brain and Language

    (2000)
  • K. Stromswold et al.

    Localization of syntactic comprehension by positron emission tomography

    Brain and Language

    (1996)
  • D.A. Balota et al.

    Visual word recognition of single-syllable words

    Journal of Experimental Psychology: General

    (2004)
  • D.A. Balota et al.

    The English Lexicon Project

    Behavior Research Methods

    (2007)
  • A. Bartels et al.

    Functional brain mapping during free viewing of natural scenes

    Human Brain Mapping

    (2004)
  • Y. Benjamini et al.

    Controlling the false discovery rate: A practical and powerful approach to multiple testing

    Journal of the Royal Statistical Society. Series B (Methodological)

    (1995)
  • Y. Benjamini et al.

    The control of the false discovery rate in multiple testing under dependency

    Annals of Statistics

    (2001)
  • M. Ben-Shachar et al.

    The neural reality of syntactic transformations: Evidence from fMRI

    Psychological Science

    (2003)
  • Bikel, D. M. (2002). Design of a multi-lingual, parallel-processing statistical parsing engine. In Proceedings of the...
  • J.R. Binder et al.

    Distinct brain systems for processing concrete and abstract concepts

    Journal of Cognitive Neuroscience

    (2005)
  • G.M. Boynton et al.

    Linear systems analysis of functional magnetic resonance imaging in human V1

    Journal of Neuroscience

    (1996)
  • Bradley, D. C. (1983). Computational distinctions of vocabulary type. Indiana University Linguistics...
  • D. Caplan et al.

    Vascular responses to syntactic processing: Event-related fMRI study of relative clauses

    Human Brain Mapping

    (2002)
  • Cited by (0)

    View full text