Explainable deep learning framework for multimodal emotion recognition using physiological signals
*Md IqbalCorresponding authoriqbal.hodcse@gmail.comDepartment of Computer Science & EngineeringQuantum School of TechnologyQuantum UniversityRoorkee, Uttarakhand, 247667, IndiaView full profile → , Rishi Kumar SharmaRishi.k.sharma@gmail.comDepartment of Computer Science & EngineeringQuantum School of TechnologyQuantum UniversityRoorkee, Uttarakhand, 247667, IndiaView full profile → , Vivek Kumarvksingh087@gmail.comDepartment of Computer Science & EngineeringQuantum School of TechnologyQuantum UniversityRoorkee, Uttarakhand, 247667, IndiaView full profile → , Arvinda Kushwahaarvindakush@gmail.comDepartment of Data SciencesGalgotia College of Engineering & Technology (GCET)Greater Noida, Uttar Pradesh, 201310, IndiaView full profile → , Hriday Kumar Guptahridaykumargupta@gmail.comDepartment of Computer Science and EngineeringGraphics Era (Deemed to be University)Dehradun, Uttrakhand, 248002, IndiaView full profile → , Mohd Anas Khananas.cse786@gmail.comDepartment of Computer Science & EngineeringSchool of EngineeringJawaharlal Nehru UniversityMehrauli, New Delhi, 110067, IndiaView full profile →
* Corresponding author · click or hover a name for details
- Received:
- 01 Jan 2026
- Published Online:
- 31 Jul 2026
- Article type:
- Research Article
- Language:
- EN
- Article no.:
- JIOS-2324
- Pages:
- 2681–2694
Abstract
Emotion recognition is one of the most essential components of affect-aware systems in human–computer interaction, mental health assessment, and adaptive intelligent environments. Despite recent progress, current methods typically also suffer from lack of coherent model integration, poor interpretability, and have limited stability and robustness in real-world constraints. Physiological signals such as electroencephalography (EEG), electrocardiography (ECG), heart rate variability (HRV), galvanic skin response (GSR), electromyography (EMG), and respiration provide objectively complementary affective signals, but their coupling with non-physiological modalities, e.g., facial expression and speech, have been less investigated from explainability-oriented points of view. Three contributions were made in this work. The first is to propose a unified interpretable multimodal emotion recognition framework that combines modality-specific feature extraction and principled fusion to jointly model heterogeneous physiological and peripheral signals. Second, it provides explainability directly in the learning architecture, so cross-modal explanations can be coherent with known physiological correlates of emotion. Third, it develops a complete evaluation framework encompassing recognition performance and explanation quality in cross-subject and cross-dataset contexts. In addition to improvements in performance, the framework integrates ethical concerns, reproducibility, and deployment viability, helping to promote reliable and transparent emotion recognition models. The following outlines further prospects for privacy-preserving learning and more advanced explanation interfaces.
Keywords
Subject Classifications
References
[1] R. W. Picard, Affective Computing. Cambridge, MA, USA: MIT Press (1997).
[2] R. A. Calvo and S. D’Mello, “Affect detection: An interdisciplinary review,” IEEE Trans. Affect. Comput., vol. 1, no. 1, pp. 18–37 (2010).
[3] S. D’Mello and J. Kory, “A review and meta-analysis of multimodal affect detection systems,” ACM Comput. Surv., vol. 47, no. 3, pp. 1–36 (2015).
[4] C. Rudin, “Stop explaining black box machine learning models for high stakes decisions,” Nat. Mach. Intell., vol. 1, pp. 206–215 (2019).
[5] W. Samek, T. Wiegand, and K. R. Müller, “Explainable artificial intelligence: Understanding, visualizing and interpreting deep learning models,” IT Prof., vol. 21, no. 6, pp. 39–48 (2019).
[6] T. Khan, K. Singh, and K. C. Purohit, “ICMA: An efficient integrated congestion control approach,” Recent Patents Eng., vol. 14, no. 3, pp. 294–309 (2020).
[7] M. Z. Soroush, K. Maghooli, S. K. Setarehdan, and A. Nasrabadi, “A review on EEG signals-based emotion recognition,” Int. Clin. Neurosci. J., vol. 4, no. 4, pp. 118–129 (2017).
[8] M. Soleymani, J. Lichtenauer, T. Pun, and M. Pantic, “Multimodal emotion recognition in response to videos,” IEEE Trans. Affect. Comput., vol. 3, no. 2, pp. 211–223 (2012).
[9] N. Sebe, I. Cohen, T. S. Huang, and T. Gevers, “Multimodal approaches for emotion recognition: A survey,” in Proc. SPIE, vol. 5670, pp. 56–67 (2005).
[10] A. Dzedzickis, A. Kaklauskas, and V. Bucinskas, “Human emotion recognition: Review of sensors and methods,” Sensors, vol. 20, no. 3, p. 592 (2020).
[11] Z. Li, F. Liu, W. Yang, S. Peng, and J. Zhou, “A survey of convolutional neural networks,” IEEE Trans. Neural Netw. Learn. Syst., vol. 33, no. 12, pp. 6999–7019 (2021).
[12] T. Baltrušaitis, C. Ahuja, and L.-P. Morency, “Multimodal machine learning: A survey and taxonomy,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 41, no. 2, pp. 423–443 (2019).
[13] R. Guidotti, A. Monreale, S. Ruggieri, F. Turini, D. Pedreschi, and F. Giannotti, “A survey of methods for explaining black box models,” ACM Comput. Surv., vol. 51, no. 5, pp. 1–42 (2018).
[14] E. Tjoa and C. Guan, “A survey on explainable artificial intelligence for time series,” IEEE Trans. Neural Netw. Learn. Syst., vol. 32, no. 11, pp. 4793–4810 (2021).
[15] L. Floridi, J. Cowls, M. Beltrametti, R. Chatila, P. Chazerand, V. Dignum, C. Luetge, R. Madelin, U. Pagallo, F. Rossi, B. Schafer, P. Valcke, and E. Vayena, “AI4People—An ethical framework for a good AI society,” Minds Mach., vol. 28, pp. 689–707 (2018).
[16] S. Koelstra, C. Muhl, M. Soleymani, J.-S. Lee, A. Yazdani, T. Ebrahimi, T. Pun, A. Nijholt, and I. Patras, “DEAP: A database for emotion analysis using physiological signals,” IEEE Trans. Affect. Comput., vol. 3, no. 1, pp. 18–31 (2012).
[17] M. Z. Soroush, K. Maghooli, S. K. Setarehdan, and A. Nasrabadi, “A review on EEG signals based emotion recognition,” Int. Clin. Neurosci. J., vol. 4, no. 4, pp. 118–129 (2017).
[18] C. Li, Z. Bao, L. Li, and Z. Zhao, “EEG-based emotion recognition using deep learning: A review,” IEEE Trans. Neural Netw. Learn. Syst., (2021).
[19] M. Soleymani, J. Lichtenauer, T. Pun, and M. Pantic, “Multimodal emotion recognition in response to videos,” IEEE Trans. Affect. Comput., vol. 3, no. 2, pp. 211–223 (2012).
[20] T. Baltrušaitis, C. Ahuja, and L.-P. Morency, “Multimodal machine learning: A survey and taxonomy,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 41, no. 2, pp. 423–443 (2019).
[21] E. Tjoa and C. Guan, “A survey on explainable artificial intelligence for time series,” IEEE Trans. Neural Netw. Learn. Syst., vol. 32, no. 11, pp. 4793–4810 (2021).
[22] R. Guidotti, A. Monreale, S. Ruggieri, F. Turini, D. Pedreschi, and F. Giannotti, “A survey of methods for explaining black box models,” ACM Comput. Surv., vol. 51, no. 5, pp. 1–42 (2018).
[23] C. Rudin, “Stop explaining black box machine learning models for high stakes decisions,” Nat. Mach. Intell., vol. 1, pp. 206–215 (2019).
[24] A. Dzedzickis, A. Kaklauskas, and V. Bucinskas, “Human emotion recognition: Review of sensors and methods,” Sensors, vol. 20, no. 3, p. 592 (2020).
[25] E. G. Han, T.-K. Kang, and M.-T. Lim, “Physiological signal-based real-time emotion recognition based on exploiting mutual information with physiologically common features,” Electronics, vol. 12, no. 13, p. 2933 (2023).
[26] S. Koelstra, C. Muhl, M. Soleymani, J.-S. Lee, A. Yazdani, T. Ebrahimi, T. Pun, A. Nijholt, and I. Patras, “DEAP: A database for emotion analysis using physiological signals,” IEEE Trans. Affect. Comput., vol. 3, no. 1, pp. 18–31 (2012).
[27] A. Dzedzickis, A. Kaklauskas, and V. Bucinskas, “Human emotion recognition: Review of sensors and methods,” Sensors, vol. 20, no. 3, p. 592 (2020).
[28] J. A. Russell, “A circumplex model of affect,” J. Pers. Soc. Psychol., vol. 39, no. 6, pp. 1161–1178 (1980).
[29] M. Soleymani, J. Lichtenauer, T. Pun, and M. Pantic, “Multimodal emotion recognition in response to videos,” IEEE Trans. Affect. Comput., vol. 3, no. 2, pp. 211–223 (2012).
[30] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” in Adv. Neural Inf. Process. Syst., vol. 30, pp. 5998–6008 (2017).
[31] T. Khan, K. Singh, A. K. Sharma, and K. C. Purohit, “An efficient trust-based decision-making approach for WSNs: Machine learning oriented approach,” Comput. Commun., vol. 209, pp. 217–229 (2023).
[32] A. Sharma, R. Verma, and S. K. Singh, “Explainable deep learning framework for healthcare data analysis using physiological signals,” J. Inf. Optim. Sci., vol. 44, no. 2, pp. 345–360 (2023).
[33] P. Gupta, M. Iqbal, and R. Kumar, “Multimodal emotion recognition using deep learning techniques: A fusion-based approach,” J. Inf. Optim. Sci., vol. 43, no. 6, pp. 1121–1135 (2022).
[34] V. Kumar and S. Kumar, “Advancing mango leaf disease detection with YOLOv11 incorporating white scale and cottony cushion scale,” J. Agric. Eng. (India), vol. 63, no. 1, pp. 1–15 (Feb. 2026).
