This Special Issue is based on presentations at the Workshop on “Action, Language and Neuroinformatics” held in July of 2011. It contributes to the view that neuroinformatics must include the informatics of computational modeling of neural systems as well as the development and linkage of database resources for both models and empirical data. The papers introduce key results in empirical research, computational modeling, and neuroinformatics for two areas of neuroscience–neurolinguistics and the study of neural mechanisms underlying manual action and its recognition–and assess ways in which the study of language processing can benefit from models of action production. Because the two areas span the spectrum from animal studies to human studies, and from basic sensorimotor processes to cognition, they provide a setting for assessing the diverse challenges of creating computational models for neuroscience, for providing databases for the very different forms of empirical data now exploited by neuroscience, and for linkage of data and models in systems and cognitive neuroscience generally, not just within our two focal areas.

The papers in this Special Issue are divided into four parts: Part 1, Databasing the Brain, presents three approaches to the development of neuroinformatics databases, including a new methodology for linking data and models in systems and cognitive neuroscience, tools for federating online neuroinformatics databases, and tools to link gene expression data to cognitive brain systems. Part 2, Action, Imitation and Gesture, provides two cases studies linking research on monkeys, apes, humans and machines, exemplifying neuroinformatics in the wide sense that embraces computational modeling as well as database construction. Part 3, Language, develops this story in relation to the uniquely human capacity for language, offering models of human syntactic encoding and decoding, actor-based language comprehension and the linkage of visual scenes to language via template construction grammar, in each case considering how to test the models against data from human behavior and brain imaging. Finally, Part 4 builds upon a series of intense discussions held at the Workshop on the present and future of neuroinformatics, with especial emphasis on the integration of computational models with empirical data.

Part 1. Databasing the Brain

Michael Arbib, Anon Plangprasopchok, James Bonaiuto & Rob Schuler: A Neuroinformatics of Brain Modeling and its Implementation in the Brain Operation Database BODB

The Brain Operation Database BODB is organized around two main ideas: (i) A brain model should be characterized not only by a structural ontology (the brain regions or finer structures to which it corresponds) but also by a functional ontology (the Brain Operating Principles which it exemplifies). (ii) Documentation of the model should link both to the empirical data used in designing it, and to the empirical data used to test it. This allows researchers to place a model of brain function within the proper context of competing models of similar systems, and the data which guide production of these models. In addition, the Brain Operating Principles which populate the functional ontology of the database provide a theoretical footing to analyze how these abstract principles are realized in brain tissue. A model can be used to anchor a search for other models which address related empirical data. BODB can then generate a table to compare the explanatory reach of these models. This benchmarking provides one of the facilities being developed for Collaboratory Workspaces that can be used to promote collaborations between modelers and experimentalists on specific topics in developing summaries of related empirical data and building models which address them.

Finn Årup Nielsen: Brede Tools and Federating Online Neuroinformatics Databases

Building on the BrainMap resource for structured access to functional neuroimaging data, the Brede Database and Brede Wiki are open source neuroinformatics databases that seek to make distribution, federation and interaction of their content as easy and transparent as possible. The philosophy behind the databases has been to rely on simple formats and allow other online tools to use their content. This paper describes the possible interconnections on different levels between the Brede tools and other databases.

Jason W. Bohland, Emma M. Myers and Esther Kim: An Informatics Approach to Integrating Genetic and Neurological Data in Speech and Language Neuroscience

The architecture of language-related brain systems can be described at multiple scales, from genes and gene products that guide brain development and provide the molecular signatures of populations of neurons to the circuits that comprise functional subsystems. While much of the neurobiologically based experimental data related to language comes from neuroimaging or lesion studies, new efforts enable connections to be made between genes and gene sets and the higher-level organization of these brain systems. The paper describes such efforts in mouse and human, and discusses a new related web database aimed at neurolinguistics researchers as well as the issue of brain atlas concordance, which is important in unifying diverse human brain datasets. Finally, it assesses the daunting challenges of linking gene expression data to models of brain mechanisms underlying language.

Part 2. Action, Imitation and Gesture

Yiannis Demiris, Lisa Aziz-Zadeh and James Bonaiuto: Information Processing in the Mirror Neuron System in Primates and Machines

The mirror neuron system in primates matches observations of actions with the motor representations used for their execution. Models of this system have contributed both to systems neuroscience and to enablin robots to imitate and learn how to perform tasks from human demonstrations. The challenge is to develop principled meta-analyses of models that differ in the granularity of their experimental setups, methodologies, computational structures and selected modeled data. The paper uses the Brain Operation Database BODB (see the paper by Arbib, Plangprasopchok, Bonaiuto & Schuler in this issue) to bring together complementary computational models by functionally decomposing them into brain operating principles (BOPs) which each capture a limited subset of the model’s functionality. The paper uses links from these BOPs to neuroimaging and neurophysiological data in order to pinpoint complementary and conflicting explanations and compare predictions against selected sets of neurobiological data. The results of this comparison are used to interpret neuroimaging results in terms of neural network activity, evaluate the biological plausibility of imitation models, and suggest new experiments that can shed light on the neural basis of imitation.

Brad Gasser, Erica Cartmill and Michael A. Arbib: Ontogenetic Ritualization of Primate Gesture as a Case Study in Dyadic Brain Modeling

The paper surveys the controversy over whether or not ape gestures are learned, but nonetheless uses the assessment of ontogenetic ritualization (whereby actions become ritualized as communicative gestures which relate to these actions) to clarify the physical interactions and neural changes that seem necessary to support the transition from action to gesture more generally. The paper charts the challenges involved in passing from separate models (each documented in BODB) of brain mechanisms governing action and action recognition in the macaque brain to an integrated model of the chimpanzee brain to be simulated within two interacting agents–thus, dyadic brain modeling–and tested against empirical data on ape gesture. The paper closes by relating this work to the neuroinformatics challenges of integrating and sharing data to support collaboration between primatologists, neuroscientists and modelers that will help speed the emergence of what may be called comparative neuro-primatology.

Part 3. Language

Gerard Kempen: Prolegomena to a Neurocomputational Architecture for Human Grammatical Encoding and Decoding

The paper models syntactic structure formation as an interactive activation and inhibition (IAC) process running in a fixed-connectivity neural network. The basic design requirement is that structure formation should subserve sentence production as well as sentence comprehension, taking as input either ordered strings of potentially ambiguous lexical items (in comprehension), or unordered sets of unambiguous dependency-related lexical items (in production). The design includes a neurocomputational mechanism for the treatment of an important class of grammatical movement phenomena. The model is evaluated in terms of its implications for explaining selected psycholinguistic and neurolinguistic phenomena.

Phillip Alday, Matthias Schlesewsky & Ina Bornkessel-Schlesewsky: Towards a Computational Model of Actor-Based Language Comprehension

The paper assesses neurophysiological data from a range of typologically diverse languages to offer evidence for a cross-linguistically valid actor-based strategy of understanding sentence-level meaning. This strategy seeks to identify the participant primarily responsible for the state of affairs (the actor) as quickly and unambiguously as possible, thus resulting in competition for the actor role when there are multiple candidates. It is proposed that this strategy may derive from more basic cognitive or neurobiological organizational principles (e.g., the human ability to recognize goal-directed action). The paper offers an initial computational model of the actor-strategy and how it interacts with language-specific properties. It tests how effectively several metrics of actor competition derived from the model predict modulations of neurophysiological activity related to sentence comprehension. The authors argue that their model can form the basis for a more comprehensive, neurobiologically plausible computational model of constructing sentence-level meaning.

Victor Barrès and Jinyong Lee: Template Construction Grammar: From Visual Scene Description to Language Comprehension and Agrammatism

The paper presents SemRep as an abstract form of visual representation with an emphasis on the spatial linkage of entities, attributes and actions, and Template Construction Grammar (TCG) as a SemRep-coupled implementation of construction grammar. In the SemRep-TCG model of utterance production while describing a visual scene, the vision system interprets a part of the scene under the current focus of attention, and then the SemRep is created or updated accordingly in the working memory. The language system applies TCG constructions on the SemRep built in the working memory The deployment mechanism of attentional focus is hypothesized to be the main driving force of the resultant sentential structure. Eye-tracking experiments provide a test of the model assumptions. Complementary work on language production, building a SemRep from an utterance, provides a dual path model involving the “heavy” semantics of world knowledge versus the “light” semantics in the syntactico-semantic categories implicit in the constructions. The model is related to data on agrammatism. Complementing the work on modeling, the strategy for entering summaries of empirical data (SEDs) in BODB for fMRI data is described, and the challenges of representing ERP data in a form suitable for testing neurolinguistic models are assessed.

Part 4. Ongoing Challenges

Michael Arbib, James Bonaiuto, Ina Bornkessel-Schlesewsky, David Kemmerer, Brian MacWhinney, Finn Årup Nielsen and Erhan Oztop: Action and Language Mechanisms in the Brain: Data, Models and Neuroinformatics

This paper summarizes the discussions held on the third day of the Workshop. It offers a broad discussion of the neuroinformatics challenges (both data collation and modeling) posed by the need to analyze and comprehend large-scale data, in both humans and other species, underlying action and language. Driving questions include: How should data be summarized for the design and testing of computational models? How should summarized data be linked to more expansive stores of empirical data? What environments can facilitate collaborations between modelers and experimentalists? The paper is framed by the claim that action and language are generic with respect to assessing the challenges of a neuroinformatics that supports both systems and cognitive neuroscience.