Adaptive semantic fusion based scene aligned segmentation for optimizing the efficacy of multimodal video classification
*Karre RathiCorresponding authorkarrerathi8@gmail.comDepartment of Computer Science & EngineeringAnnamalai UniversityAnnamalainagar, Tamil Nadu, 608002, IndiaView full profile → , T. Sasi Roobasasiruba@gmail.comDepartment of Computer Science & EngineeringAnnamalai UniversityAnnamalainagar, Tamil Nadu, 608002, IndiaView full profile → , K. Subba Reddymrsubbareddy@yahoo.comDepartment of Computer Science and EngineeringRajeev Gandhi Memorial College of Engineering and TechnologyNandyal, Andhra Pradesh, 518501, IndiaView full profile →
* Corresponding author · click or hover a name for details
- Received:
- 01 Mar 2026
- Published Online:
- 31 Oct 2026
- Article type:
- Research Article
- Language:
- EN
- Article no.:
- JIOS-2485
- Pages:
- 3575–3584
Abstract
Multimodal video segmentation and classifications are an emerging need for the detection of video-based fraudulent applications. Existing fusion methods, such as the Multimodal Embedding Alignment System (MEAS), Transformer models, and Visual Semantic Multi-Head Attention (VSMHA), have produced prominent multimodal video classification results. However, they rely on static segmentation and uniform fusion strategies, which weakens cross-modal interaction and leads to the loss of contextual information. Nevertheless, they rely on uniform fusion and static segmentation, which weaken crossmodal interactions and lead to the loss of relevant context. To address these limitations, this paper presents a Dynamic Adaptive Semantic Fusion Framework (DASFF) that integrates an adaptive video segmentation system that synchronises semantic scene changes with segment boundaries in real time. The proposed method enables more coherent and discriminative feature representations by capturing both temporal dynamics and contextual semantics. To demonstrate the efficacy of the proposed DASFF method, an extensive experimental study is conducted using the TAU urban audio-visual dataset. The proposed DASFF achieved an accuracy of 98.70% and demonstrated robust improvements in statistical performance. The results indicate the usefulness of semantic-aware fusion and adaptive segmentation for improved multimodal video classifications.
Keywords
Subject Classifications
References
[1] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” in Adv. Neural Inf. Process. Syst. (NeurIPS), vol. 30, pp. 5998–6008 (2017).
[2] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov, “RoBERTa: A robustly optimized BERT pretraining approach,” arXiv:1907.11692 (2019), doi: 10.48550/arXiv.1907.11692.
[3] T. Baltrušaitis, C. Ahuja, and L.-P. Morency, “Multimodal machine learning: A survey and taxonomy,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 41, no. 2, pp. 423–443 (2019), doi: 10.1109/TPAMI.2018.2798607.
[4] A. Zadeh, M. Chen, S. Poria, E. Cambria, and L.-P. Morency, “Tensor fusion network for multimodal sentiment analysis,” in Proc. Conf. Empirical Methods Nat. Lang. Process. (EMNLP), pp. 1103–1114 (2017).
[5] C.-Y. Wu, S. Zaheer, H. Hu, R. Manmatha, M. Smolić, and P. Krähenbühl, “Compressed video action recognition,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), pp. 6026–6035 (2018).
[6] L. Zhou, Y. Zhou, X. Qi, J. Hu, T. L. Lam, and Y. Xu, “Feature pyramid attention network for audio-visual scene classification,” CAAI Trans. Intell. Technol., vol. 10, no. 2, pp. 359–374 (2025), doi: 10.1049/cit2.12375.
[7] H. Wang, Z.-J. Zha, L. Li, X. Chen, and J. Luo, “Context-aware proposal-boundary network with structural consistency for audiovisual event localization,” IEEE Trans. Neural Netw. Learn. Syst., vol. 35, no. 11, pp. 15872–15882 (2024), doi: 10.1109/TNNLS.2023.3290083.
[8] M. B. Shaikh, D. Chai, S. M. S. Islam, and N. Akhtar, “Multimodal fusion for audio-image and video action recognition,” Neural Comput. Appl., vol. 36, pp. 5499–5513 (2024), doi: 10.1007/s00521-023-09186-5.
[9] B. Gayathri and R. Gopi, “Some properties of k-even mean graph,” J. Discrete Math. Sci. Cryptogr., vol. 29, no. 4, pp. 1579–1583 (2025), doi: 10.47974/JDMSC-2244.
[10] S. Gong, Y. Zhuge, L. Zhang, Y. Wang, P. Zhang, L. Wang, and H. Lu, “AVS-Mamba: Exploring temporal and multi-modal Mamba for audio-visual segmentation,” IEEE Trans. Multimedia, vol. 27, pp. 5413–5425 (2025), doi: 10.1109/TMM.2025.3542995.
[11] Q. Wei, Y. Zhou, S. Xiang, L. Xiao, and Y. Zhang, “MEAS: Multimodal emotion analysis system for short videos on social media platforms,” IEEE Trans. Comput. Social Syst., vol. 12, no. 5, pp. 2398–2410 (2025), doi: 10.1109/TCSS.2024.3490846.
[12] A. Mesaros, T. Heittola, and T. Virtanen, “TAU Urban Audio-Visual Scenes 2021 dataset,” in Proc. Detection Classification Acoust. Scenes Events (DCASE), Tampere Univ. (2021).
[13] S. Gong, Y. Zhuge, L. Zhang, P. Zhang, and H. Lu, “Complementary and contrastive learning for audio-visual segmentation,” IEEE Trans. Multimedia, vol. 27, pp. 7407–7420 (2025).
[14] B. Mocanu, R. Tapu, and T. Zaharia, “Multimodal emotion recognition using cross modal audio-video fusion with attention and deep metric learning,” Image Vis. Comput., vol. 133, Art. no. 104676 (2023), doi: 10.1016/j.imavis.2023.104676.
[15] C. Nalini, C. Raghavendra, and K. Rajendra Prasad, “Comparative observation and performance analysis of multiple algorithms on Iris data,” Int. J. Pure Appl. Math., vol. 116, no. 9, pp. 319–325 (2017).
[16] C. Leelavathi, G. Thahniyath, and K. Rajendra Prasad, “Optimizing the deep features using the spectral concept for an efficient video classification,” J. Inf. Optim. Sci., vol. 46, no. 2, pp. 553–562 (2025).
[17] M. Iqbal, R. K. Sharma, V. Kumar, A. Kushwaha, H. K. Gupta, and M. A. Khan, “Explainable deep learning framework for multimodal emotion recognition using physiological signals,” J. Inf. Optim. Sci., vol. 47, no. 7, pp. 2681–2694 (2026), doi: 10.47974/JIOS-2324.
