Emotion recognition is one of the most essential components of affect-aware systems in human–computer interaction, mental health assessment, and adaptive intelligent environments. Despite recent progress, current methods typically also suffer from lack of coherent model integration, poor interpretability, and have limited stability and robustness in real-world constraints. Physiological signals such as electroencephalography (EEG), electrocardiography (ECG), heart rate variability (HRV), galvanic skin response (GSR), electromyography (EMG), and respiration provide objectively complementary affective signals, but their coupling with non-physiological modalities, e.g., facial expression and speech, have been less investigated from explainability-oriented points of view. Three contributions were made in this work. The first is to propose a unified interpretable multimodal emotion recognition framework that combines modality-specific feature extraction and principled fusion to jointly model heterogeneous physiological and peripheral signals. Second, it provides explainability directly in the learning architecture, so cross-modal explanations can be coherent with known physiological correlates of emotion. Third, it develops a complete evaluation framework encompassing recognition performance and explanation quality in cross-subject and cross-dataset contexts. In addition to improvements in performance, the framework integrates ethical concerns, reproducibility, and deployment viability, helping to promote reliable and transparent emotion recognition models. The following outlines further prospects for privacy-preserving learning and more advanced explanation interfaces.