TARU PUBLICATIONS
Journal of Interdisciplinary Mathematics cover
Open Access ·Peer-reviewed·ISSN (Online): 2169-012X·ISSN (Print): 0972-0502

Monthly Journal: Publishes the methodological and theoretical role of mathematics and mathematical applications underpinning scientific research.

Issues up to 2022 co-published with and available at:Taylor & Francis Online
submissions@tarupublications.com
Open Access Research Article

Quantitative fusion of NLP, fMRI, and EEG data : A mathematical model for decoding semantic processing in the brain

, , , * ,

* Corresponding author · click or hover a name for details

pp. 403–414Vol. 27Issue 2March 2024DOI: 10.47974/JIM-1891XML
Published Online:
19 Mar 2024
Article type:
Research Article
Language:
EN
Article no.:
JIM-1891
Pages:
403–414

Abstract

This research introduces a novel mathematical model designed to integrate Natural Language Processing (NLP), functional Magnetic Resonance Imaging (fMRI), and Electroencephalography (EEG) data, aiming to decode the complex neural mechanisms of semantic processing in the human brain. By leveraging the complementary strengths of each modality—NLP’s linguistic analysis, fMRI’s spatial resolution, and EEG’s temporal precision—the model provides a groundbreaking approach to understanding how semantic information is processed across different brain regions and over time. The core of the proposed model is a dynamic, multi-layered framework that utilizes advanced statistical methods and machine learning algorithms. At its foundation, the model employs vector space representations from NLP to quantify semantic similarity and contextuality in language. These representations are then mapped onto neural activation patterns captured by fMRI and EEG, using a series of transformation matrices that are optimized through machine learning techniques. The model uniquely incorporates time-series analysis to account for the temporal dynamics of EEG data, while spatial patterns from fMRI data are analyzed through convolutional neural networks, ensuring a comprehensive integration of multimodal neuroimaging data. Key to proposed approach is the application of Bayesian inference methods to fuse these diverse data sources, allowing for the probabilistic modeling of semantic processing pathways in the brain. This enables the prediction of neural responses to linguistic stimuli with unprecedented accuracy and detail. Theoretical implications of our model suggest significant advances in understanding the neural basis of language comprehension, offering new insights into the dynamic interplay between linguistic structures and neural processes.

Keywords

Subject Classifications

05C10

References

[1] Hasson, U., Nusbaum, H., & Duncan, J. Multimodal integration of neuroimaging data in cognitive neuroscience. Nature Neuroscience, pages 1353-1362. (2020).
[2] Bartolomeo, P., & Merchant, H. Bridging the gap between language and action: Integrating NLP and cognitive neuroscience. Language and Linguistics Compass, pages e12650. (2022).
[3] Pouget-Chauvet, M., Dayan, P., & Daunizeau, J. Deep learning for the integration of neuroimaging and behavioral data in cognitive neuroscience. Nature Neuroscience, pages 1426-1434. (2020).
[4] Nguyen, M., Garnotel, R., Tomoto, H., & Moreau, Q. Multimodal machine learning for mapping language to the brain: A survey. NeuroImage, pages 117231. (2020).
[5] Schreiber, A., Rokem, A., & Leahu, R. Deep learning for integrating neuroimaging and language data: A review of methods and applications. Journal of Neuroscience Methods, pages 108682. (2022).
[6] Blankenship, J., & Bandettini, P. A. The role of magnetic resonance imaging (MRI) in understanding the neurobiology of language. Cortex, pages 803-820. (2010).
[7] Sun, Y., et al. A deep learning framework for multimodal fusion of EEG, fMRI, and behavioral data for semantic processing. (2023).
[8] Kim, Y., et al. Deep multi-modal learning for joint analysis of EEG and fMRI in semantic processing. (2022).
[9] Mumford, J. A., et al. Multivariate pattern analysis of fMRI data. (2022).
[10] Michel, C. M., & Koenig, T. EEG microstates: A window into the brain’s dynamic instabilities. (2022).
[11] Zhang, Lipeng, Peng Zhang, Xindian Ma, Shuqin Gu, Zhan Su, and Dawei Song. “A generalized language model in tensor space.” In Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, no. 01, pp. 7450-7458. (2019).
[12] Doyle, Cameron M. “Unsupervised Classification Reveals Degenerate Neural Representations of Emotion.” (2020).

Views: 411Downloads: 133Citations: 0