Quantitative fusion of NLP, fMRI, and EEG data : A mathematical model for decoding semantic processing in the brain
Vivek Kumar Vermavermavivek123@gmail.comDepartment of Information TechnologyManipal University JaipurJaipur, Rajasthan, IndiaView full profile → , Gottumukkala K. Chaitanya Varmagkcv16@gmail.comDepartment of Information TechnologyManipal University JaipurJaipur, Rajasthan, IndiaView full profile → , Ch. Karthikeya Varmakarthikeyavarmackv7@gmail.comDepartment of Information TechnologyManipal University JaipurJaipur, Rajasthan, IndiaView full profile → , *Anita ShrotriyaCorresponding authoranitashrotriya@gmail.comDepartment of Information TechnologyManipal University JaipurJaipur, Rajasthan, IndiaView full profile → , Pankaj Dadheechpankajdadheech777@gmail.comDepartment of Computer Science and EngineeringSwami Keshvanand Institute of Technology, Management & GramothanJaipur, Rajasthan, IndiaView full profile →
* Corresponding author · click or hover a name for details
- Published Online:
- 19 Mar 2024
- Article type:
- Research Article
- Language:
- EN
- Article no.:
- JIM-1891
- Pages:
- 403–414
Abstract
Keywords
Subject Classifications
References
[1] Hasson, U., Nusbaum, H., & Duncan, J. Multimodal integration of neuroimaging data in cognitive neuroscience. Nature Neuroscience, pages 1353-1362. (2020).
[2] Bartolomeo, P., & Merchant, H. Bridging the gap between language and action: Integrating NLP and cognitive neuroscience. Language and Linguistics Compass, pages e12650. (2022).
[3] Pouget-Chauvet, M., Dayan, P., & Daunizeau, J. Deep learning for the integration of neuroimaging and behavioral data in cognitive neuroscience. Nature Neuroscience, pages 1426-1434. (2020).
[4] Nguyen, M., Garnotel, R., Tomoto, H., & Moreau, Q. Multimodal machine learning for mapping language to the brain: A survey. NeuroImage, pages 117231. (2020).
[5] Schreiber, A., Rokem, A., & Leahu, R. Deep learning for integrating neuroimaging and language data: A review of methods and applications. Journal of Neuroscience Methods, pages 108682. (2022).
[6] Blankenship, J., & Bandettini, P. A. The role of magnetic resonance imaging (MRI) in understanding the neurobiology of language. Cortex, pages 803-820. (2010).
[7] Sun, Y., et al. A deep learning framework for multimodal fusion of EEG, fMRI, and behavioral data for semantic processing. (2023).
[8] Kim, Y., et al. Deep multi-modal learning for joint analysis of EEG and fMRI in semantic processing. (2022).
[9] Mumford, J. A., et al. Multivariate pattern analysis of fMRI data. (2022).
[10] Michel, C. M., & Koenig, T. EEG microstates: A window into the brain’s dynamic instabilities. (2022).
[11] Zhang, Lipeng, Peng Zhang, Xindian Ma, Shuqin Gu, Zhan Su, and Dawei Song. “A generalized language model in tensor space.” In Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, no. 01, pp. 7450-7458. (2019).
[12] Doyle, Cameron M. “Unsupervised Classification Reveals Degenerate Neural Representations of Emotion.” (2020).




