Abstract
The dispute of how the human brain represents conceptual knowledge has been argued in many scientific fields. Brain imaging studies have shown that the spatial patterns of neural activation in the brain are correlated with thinking about different semantic categories of words (for example, tools, animals, and buildings) or when viewing the related pictures. In this paper, we present a computational model that learns to predict the neural activation captured in functional magnetic resonance imaging (fMRI) data of test words. Unlike the models with hand-crafted features that have been used in the literature, in this paper we propose a novel approach wherein decoding models are built with features extracted from popular linguistic encodings of Word2Vec, GloVe, Meta-Embeddings in conjunction with the empirical fMRI data associated with viewing several dozen concrete nouns. We compare these models with several other models that use word features extracted from FastText, Randomly-generated features, Mitchell’s 25 features. The experimental results show that the predicted fMRI images using Meta-Embeddings meet the state-of-the-art performance. Although models with features from GloVe and Word2Vec predict fMRI images similar to the state-of-the-art model, model with features from Meta-Embeddings predicts significantly better. The proposed scheme that uses popular linguistic encoding offers a simple and easy approach for semantic decoding from fMRI experiments.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
Available at http://www.cs.cmu.edu/~fmri/science2008/data.html.
References
Caramazza, A., Mahon, B.Z.: The organization of conceptual knowledge: the evidence from category-specific semantic deficits. Trends Cogn. Sci. 7(8), 354–361 (2003)
Mahon, B.Z., Caramazza, A.: What drives the organization of object knowledge in the brain? Trends Cogn. Sci. 15(3), 97–103 (2011)
Tong, F., Pratte, M.S.: Decoding patterns of human brain activity. Annu. Rev. Psychol. 63, 483–509 (2012)
Clark, V.P., Maisog, J.M., Haxby, J.V.: fMRI study of face perception and memory using random stimulus sequences. J. Neurophysiol. 79(6), 3257–3265 (1998)
Carlson, T.A., Schrater, P., He, S.: Patterns of activity in the categorical representations of objects. J. Cogn. Neurosci. 15(5), 704–717 (2003)
Howell, D.C.: Statistical Methods for Psychology. Cengage Learning, Belmont (2012)
Haxby, J.V., Gobbini, I.M., Furey, M.L., Ishai, A., Schouten, J.L., Pietrini, P.: Distributed and overlapping representations of faces and objects in ventral temporal cortex. Science 293(5539), 2425–2430 (2001)
Ishai, A., Ungerleider, L.G., Martin, A., Schouten, J.L., Haxby, J.V.: Distributed representation of objects in the human ventral visual pathway. Proc. Natl. Acad. Sci. 96(16), 9379–9384 (1999)
Cox, D.D., Savoy, R.L.: Functional magnetic resonance imaging (fMRI) “brain reading”: detecting and classifying distributed patterns of fMRI activity in human visual cortex. Neuroimage 19(2), 261–270 (2003)
Polyn, S.M., Natu, V.S., Cohen, J.D., Norman, K.A.: Category-specific cortical activity precedes retrieval during memory search. Science 310(5756), 1963–1966 (2005)
Caramazza, A., Shelton, J.R.: Domain-specific knowledge systems in the brain: the animate-inanimate distinction. J. Cogn. Neurosci. 10(1), 1–34 (1998)
Crutch, S.J., Warrington, E.K.: Spatial coding of semantic information: knowledge of country and city names depends on their geographical proximity. Brain 126(8), 1821–1829 (2003)
Samson, D., Pillon, A.: Orthographic neighborhood and concreteness effects in the lexical decision task. Brain Lang. 91(2), 252–264 (2004)
Cree, G.S., McRae, K.: Analyzing the factors underlying the structure and computation of the meaning of chipmunk, cherry, chisel, cheese, and cello (and many other such concrete nouns). J. Exp. Psychol. Gen. 132(2), 163 (2003)
Mahon, B.Z., Caramazza, A.: The orchestration of the sensory-motor systems: clues from neuropsychology. Cogn. Neuropsychol. 22(3–4), 480–494 (2005)
Miller, G.A., Beckwith, R., Fellbaum, C., Gross, D., Miller, K.J.: Introduction to wordnet: an on-line lexical database. Int. J. Lexicogr. 3(4), 235–244 (1990)
Kipper-Schuler, K.: VerbNet: a broad-coverage, comprehensive verb lexicon. Ph.D. thesis, University of Pennsylvania (2005)
Navigli, R., Ponzetto, P.S.: BabelNet: building a very large multilingual semantic network. In: Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pp. 216–225 (2010)
Mitchell, T.M., et al.: Predicting human brain activity associated with the meanings of nouns. Science 320(5880), 1191–1195 (2008)
Singh, V., Miyapuram, K.P., Bapi, R.S.: Detection of cognitive states from fMRI data using machine learning techniques. In: Proceedings of the 20th International Joint Conference on Artificial Intelligence, pp. 587–592 (2007)
Hinton, G.E., Mcclelland, J.L., Rumelhart, D.E.: Distributed representations, parallel distributed processing: explorations in the microstructure of cognition, Volume 1: Foundations (1986)
Turney, P.D., Pantel, P.: From frequency to meaning: vector space models of semantics. J. Artif. Intell. Res. 37, 141–188 (2010)
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., Dean, J.: Distributed representations of words and phrases and their compositionality. In: Advances in Neural Information Processing Systems, pp. 3111–3119 (2013)
Bojanowski, P., Grave, E., Joulin, A., Mikolov, T.: Enriching word vectors with subword information. arXiv preprint arXiv:1607.04606 (2016)
Pennington, J., Socher, R., Manning, C.: GloVe: global vectors for word representation. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1532–1543 (2014)
Yin, W., Schütze, H.: Learning meta-embeddings by using ensembles of embedding sets. arXiv preprint arXiv:1508.04257 (2015)
Abnar, S., Ahmed, R., Mijnheer, M., Zuidema, W.: Experiential, distributional and dependency-based word embeddings have complementary roles in decoding brain activity. In: Proceedings of the 8th Workshop on Cognitive Modeling and Computational Linguistics (CMCL 2018), pp. 57–66 (2018)
Pereira, F., et al.: Toward a universal decoder of linguistic meaning from brain activation. Nat. Commun. 9(1), 963 (2018)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer Nature Switzerland AG
About this paper
Cite this paper
Oota, S.R., Manwani, N., Bapi, R.S. (2018). fMRI Semantic Category Decoding Using Linguistic Encoding of Word Embeddings. In: Cheng, L., Leung, A., Ozawa, S. (eds) Neural Information Processing. ICONIP 2018. Lecture Notes in Computer Science(), vol 11303. Springer, Cham. https://doi.org/10.1007/978-3-030-04182-3_1
Download citation
DOI: https://doi.org/10.1007/978-3-030-04182-3_1
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-04181-6
Online ISBN: 978-3-030-04182-3
eBook Packages: Computer ScienceComputer Science (R0)