Study on scene classification with unimodal and multimodal deep optimization learning techniques for visual-acoustic data
*Chaitanya NukalaCorresponding authorchaitu.anju@gmail.comDepartment of Computer Science and EngineeringJawaharlal Nehru Technological University KakinadaKakinada, Andhra Pradesh, 533003, IndiaView full profile → , V. Raghavendranraghuchv@yahoo.comDepartment of Information TechnologyAditya UniversitySurampalem, Andhra Pradesh, 533437, IndiaView full profile →
* Corresponding author · click or hover a name for details
- Received:
- 01 Mar 2026
- Published Online:
- 31 Oct 2026
- Article type:
- Research Article
- Language:
- EN
- Article no.:
- JIOS-2483
- Pages:
- 3555–3564
Abstract
Scene classification is an emerging need in social applications, such as surveillance security and autonomous actions, in which unimodal learning techniques are commonly used to classify scenes from either audio or visual data. Researchers are advancing advances to develop more accurate scene classifications using multimodal techniques. Here, two modalities of data, i.e., both visual and acoustic, are learned for cross-modal fusion scene classification. This paper presents unimodal learning techniques with either acoustic or visual modalities, and the state-of-the-art in multimodal learning techniques for a comparative study of scene classification results. In unimodal, visual-based learning, scene classification is achieved as well as with acoustic-based learning techniques. The experimental study conducted using publicly available benchmarked datasets, TAU datasets, its results demonstrated that recent multimodal learning techniques, Attention graph convolutional network (AGCN), Context-Aware Proposal–Boundary Network (CAPB), Audio-Visual Semantic Modelling (AVSM), gain higher performance values than other existing techniques.
Keywords
Subject Classifications
References
[1] S. Hershey, S. Chaudhuri, D. P. W. Ellis, J. F. Gemmeke, A. Jansen, R. C. Moore, M. Plakal, D. Platt, R. A. Saurous, B. Seybold, M. Slaney, R. J. Weiss, and K. Wilson, “CNN Architectures for Large-Scale Audio Classification,” in Proc. 2017 IEEE Int. Conf. Acoust., Speech Signal Process. (ICASSP), New Orleans, LA, USA, pp. 131–135 (2017), doi: 10.1109/ICASSP.2017.7952132.
[2] J. F. Gemmeke, D. P. W. Ellis, D. Freedman, A. Jansen, W. Lawrence, R. C. Moore, M. Plakal, and M. Ritter, “Audio Set: An Ontology and Human-Labeled Dataset for Audio Events,” in Proc. 2017 IEEE Int. Conf. Acoust., Speech Signal Process. (ICASSP), New Orleans, LA, USA, pp. 776–780 (2017), doi: 10.1109/ICASSP.2017.7952261.
[3] L. Zhou, Y. Zhou, X. Qi, J. Hu, T. L. Lam, and Y. Xu, “Attentional Graph Convolutional Network for Structure-Aware Audiovisual Scene Classification,” IEEE Trans. Instrum. Meas., vol. 72, pp. 1–15 (2023), doi: 10.1109/TIM.2023.3260282.
[4] H. Wang, Z.-J. Zha, L. Li, X. Chen, and J. Luo, “Context-Aware Proposal–Boundary Network with Structural Consistency for Audiovisual Event Localization,” IEEE Trans. Neural Netw. Learn. Syst., vol. 35, no. 11, pp. 15872–15882 (2024), doi: 10.1109/TNNLS.2023.3290083.
[5] X. Pang, L. Gao, and L. Zhang, “A Video Summary Generation Model Based on Hybrid Attention Using Multimodal Features,” IEEE Access, vol. 13, pp. 37078–37095 (2025), doi: 10.1109/ACCESS.2025.3545113.
[6] L. Zhou, Y. Zhou, X. Qi, J. Hu, T. L. Lam, and Y. Xu, “Feature Pyramid Attention Network for Audio-Visual Scene Classification,” CAAI Trans. Intell. Technol., vol. 10, no. 2, pp. 359–374 (2025), doi: 10.1049/cit2.12375.
[7] M. Wang, C. Chen, Y. Xie, H. Chen, Y. Liu, and P. Zhang, “Audio-Visual Scene Classification Using Transfer Learning and Hybrid Fusion Strategy,” DCASE2021 Challenge, Tech. Rep. (Jun. 2021).
[8] M. Tan and Q. V. Le, “EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks,” in Proc. 36th Int. Conf. Mach. Learn. (ICML), vol. 97, pp. 6105–6114 (2019).
[9] D. Hu, X. Li, L. Mou, P. Jin, D. Chen, L. Jing, X. Zhu, and D. Dou, “Cross-Task Transfer for Geotagged Audiovisual Aerial Scene Recognition,” in Computer Vision—ECCV 2020, Lecture Notes in Computer Science, pp. 68–84 (2020), doi: 10.1007/978-3-030-58586-0_5.
[10] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever, “Learning Transferable Visual Models from Natural Language Supervision,” in Proc. 38th Int. Conf. Mach. Learn., vol. 139, pp. 8748–8763 (2021).
[11] Y. Hou, B. Kang, and D. Botteldooren, “Audio-Visual Scene Classification via Contrastive Event-Object Alignment and Semantic-Based Fusion,” in Proc. 2022 IEEE 24th Int. Workshop Multimedia Signal Process. (MMSP), Shanghai, China, pp. 1–6 (2022), doi: 10.1109/MMSP55362.2022.9949447.
[12] B. Mocanu, R. Tapu, and T. Zaharia, “Multimodal Emotion Recognition Using Cross-Modal Audio-Video Fusion with Attention and Deep Metric Learning,” Image Vis. Comput., vol. 133, Art. no. 104676 (2023), doi: 10.1016/j.imavis.2023.104676.
[13] M. B. Shaikh, D. Chai, S. M. S. Islam, and N. Akhtar, “Multimodal Fusion for Audio-Image and Video Action Recognition,” Neural Comput. Appl., vol. 36, pp. 5499–5513 (2024), doi: 10.1007/s00521-023-09186-5.
[14] S. Gong, Y. Zhuge, L. Zhang, P. Zhang, and H. Lu, “Complementary and Contrastive Learning for Audio-Visual Segmentation,” IEEE Trans. Multimedia, vol. 27, pp. 7407–7418 (2025), doi: 10.1109/TMM.2025.3599048.
[15] S. Gong, Y. Zhuge, L. Zhang, Y. Wang, P. Zhang, L. Wang, and H. Lu, “AVS-Mamba: Exploring Temporal and Multi-Modal Mamba for Audio-Visual Segmentation,” IEEE Trans. Multimedia, vol. 27, pp. 5413–5425 (2025), doi: 10.1109/TMM.2025.3542995.
[16] M. S. Basha, K. R. Prasad, S. K. Mouleeswaran, R. C. Poonia, and S. Sebastian, “Multi-Disease Detection System with X-Ray Images Using Deep Learning Techniques,” J. Inf. Optim. Sci., vol. 45, no. 5, pp. 1379–1388 (2024).
[17] Y. Wang, X. Qian, and W. Zhou, “Transformer-Prompted Network: Efficient Audio-Visual Segmentation via Transformer and Prompt Learning,” IEEE Signal Process. Lett., vol. 32, pp. 516–520 (2025), doi: 10.1109/LSP.2024.3524120.
[18] M. R. Ahmed, R. Haque, S. M. A. Rahman, A. W. Reza, N. Siddique, and H. Wang, “Vision-Audio Multimodal Object Recognition Using Hybrid and Tensor Fusion Techniques,” Inf. Fusion, vol. 126, Art. no. 103667 (2026), doi: 10.1016/j.inffus.2025.103667.
[19] A. Mesaros, T. Heittola, and T. Virtanen, “TAU Urban Audio-Visual Scenes 2021, Development Dataset,” Zenodo (Feb. 26, 2021), doi: 10.5281/zenodo.4477542.
[20] C. Leelavathi, G. Thahniyath, and K. R. Prasad, “Optimizing the Deep Features Using the Spectral Concept for an Efficient Video Classification,” J. Inf. Optim. Sci., vol. 46, no. 2, pp. 553–562 (2025), doi: 10.47974/JIOS-1959.
