Open Access
Research Article
Optimized multi-modal deep learning for integrated safety compliance monitoring : PPE detection, face recognition, and trade classification in industrial environments
Megha Nainmegha.201025603@muj.manipal.eduDepartment of Computer Applications Manipal University JaipurJaipur, Rajasthan, 303007, IndiaView full profile → , *Shilpa SharmaCorresponding authorshilpa.sharma@jaipur.manipal.eduDepartment of Computer Applications Manipal University JaipurJaipur, Rajasthan, 303007, IndiaView full profile → , Sandeep Chaurasiasandeep.chaurasia@jaipur.manipal.eduDepartment of Computer Science & Engineering Manipal University JaipurDepartment of Computer Science & Engineering Manipal University Jaipur Jaipur Rajasthan IndiaJaipur, Rajasthan, 303007, IndiaView full profile →
* Corresponding author · click or hover a name for details
- Received:
- 01 Apr 2025
- Published Online:
- 01 May 2026
- Article type:
- Research Article
- Language:
- EN
- Article no.:
- JIOS-2252
- Pages:
- 1623–1631
Abstract
Compliance monitoring of industrial safety remains a major challenge, due to the inherent flaws in single-modality schemes of implementing the highly multidimensional characteristics of workplace hazards. This paper aims to implement a multi-modal deep-learning system that can detect and identify both personal protective equipment (PPE) and facial features and classify and identify the trade to automatically check compliance with safety standards in the industry. The framework uses Detectron2 to get 82.33% on instance segmentation and bounding-box regression, 95.35% on facial recognition with pre-trained embeddings, 92% on a trade classifier based on a convolutional network, and 99.5% on tool detection with YOLO. These modules are merged to form the combined pipeline, which verifies the identity of the workers, verifies certified trade and ensures the use of PPE in real time and then overlaid the safety-status annotations upon visual results. The quantitative results show significant advantage over the existing techniques, such as the capacity to surpass the Average Precision of YOLOv3 by 72.3% in PPE detection. The work is innovative as it includes the multi-modal data in a holistic manner and effectively deals with the interdependencies of the determinants of safety compliance better than the single module models. This study, therefore, offers a scalable, optimized, data-driven model that provided the incidence of workplace accidents through automated surveillance, and where the operability of the system is ensured, by qualitative analyses in which the system is stable in identifying non-compliance situations such as a missing mask or vest, and in all modalities, the model is highly accurate.
Keywords
Subject Classifications
68T0768T4568T1068U10
References
[1] N. D. Nath, A. H. Behzadan, and S. G. Paal, “Deep learning for site safety: Real-time detection of personal protective equipment,” Autom. Constr., vol. 112 (Apr. 2020), doi: 10.1016/j.autcon.2020.103085.[2] X. Yang, Y. Yu, S. Shirowzhan, S. M. E. Sepasgozar, and H. Li, “Automated PPE-Tool pair check system for construction safety using smart IoT,” Journal of Building Engineering (2020), doi: 10.1016/J.JOBE.2020.101721.[3] M. I. B. Ahmed, L. Saraireh, A. Rahman, S. Al-Qarawi, A. Mhran, J. Al-Jalaoud, D. Al-Mudaifer, F. Al-Haidar, D. AlKhulaifi, M. Youldash, and M. Gollapalli, “Personal Protective Equipment Detection: A Deep-Learning-Based Sustainable Approach: A Deep-Learning-Based Sustainable Approach,” Sustainability, (2023), doi: 10.3390/SU151813990.[4] L. Hallonqvist and M. Cromsjö, “Detection of safety equipment in the manufacturing industry using image recognition,” Master’s thesis, Dept. Comput. Inf. Sci., Linköping University, Linköping, Sweden (2021)[5] J. Al-Dulaimi, J. Cosmas, and M. Abbod, “Smart Health and Safety Equipment Monitoring System for Distributed Workplaces,” De Computis (2019), doi: 10.3390/COMPUTERS8040082.[6] A. Stojadinović, N. Stojanović, and L. Stojanović, “Dynamic monitoring for improving worker safety at the workplace: use case from a manufacturing shop floor,” DEBS (2015), doi: 10.1145/2675743.2771881.[7] A. M. Vukićević, M. Petrović, P. Milošević, A. Peulić, K. Jovanović, and A. Novaković, “A systematic review of computer vision-based personal protective equipment compliance in industry practice: advancements, challenges and future directions,” Artif. Intell. Rev. (2024), doi: 10.1007/S10462-024-10978-X.[8] V. S. K. Delhi, R. Sankarlal, and A. Thomas, “Detection of Personal Protective Equipment (PPE) Compliance on Construction Site Using Computer Vision Based Deep Learning Techniques,” Front. Built Environ., vol. 6 (Sep. 2020), doi: 10.3389/fbuil.2020.00136.[9] M. Nain, S. Sharma, and S. Chaurasia, “Authentication control system for the efficient detection of hard-hats using deep learning algorithms,” Journal of Discrete Mathematical Sciences and Cryptography, vol. 24, no. 8, pp. 2291–2306 (2021), doi: 10.1080/09720529.2021.2011109.[10] N. D. Nath, A. H. Behzadan, and S. G. Paal, “Deep learning for site safety: Real-time detection of personal protective equipment,” Autom. Constr. (2020), doi: 10.1016/J.AUTCON.2020.103085.[11] M. Ali, A. Diwan, and D. Kumar, “Attendance System Optimization through Deep Learning Face Recognition,” International Journal of Computing and Digital Systems (2024), doi: 10.12785/IJCDS/1501108.[12] M. Nain, S. Sharma, and S. Chaurasia, “Computer vision based human action recognition of construction workers,” in AIP Conference Proceedings, American Institute of Physics (Jan. 2025). doi: 10.1063/5.0247833.[13] M. Nain, S. Sharma, and S. Chaurasia, “Deep Learning-Based Safety Assurance of Construction Workers: Real-Time Safety Kit Detection,” in Innovations in Computational Intelligence and Computer Vision, S. Roy, D. Sinwar, N. Dey, T. Perumal, and J. M. R. S. Tavares, Eds., Singapore: Springer Nature Singapore, pp. 279–293 (2023).[14] M. Nain, S. Sharma, and S. Chaurasia, “Safety and Compliance Management System Using Computer Vision and Deep Learning,” IOP Conf. Ser. Mater. Sci. Eng., vol. 1099, no. 1, p. 012013 (Mar. 2021), doi: 10.1088/1757-899x/1099/1/012013.[15] M. Nain, S. Sharma, and S. Chaurasia, “Pandemic Management Using Artificial Intelligence-Based Safety Measures,” in Advances in Medical Technologies and Clinical Practice, pp. 86–110 (2021). doi: 10.4018/978-1-7998-7188-0.ch007.[16] M. Nain, S. Sharma, and S. Chaurasia, “Deep Learning Oriented Smart Constructional Material and Face Recognition System for the Site Safety,” in 2024 IEEE 11th Uttar Pradesh Section International Conference on Electrical, Electronics and Computer Engineering (UPCON 2024), Institute of Electrical and Electronics Engineers Inc. (2024). doi: 10.1109/UPCON62832.2024.1098302[17] S. Sharma, L. Raja, V. Bhatnagar, D. Sharma, S. N. Bhagirath, and R. C. Poonia, “Hybrid HOG-SVM encrypted face detection and recognition model,” Journal of Discrete Mathematical Sciences and Cryptography, vol. 25, no. 1, pp. 205–218 (2022). DOI: 10.1080/09720529.2021.2014141[18] N. Kumar and D. Bhargava, “A scheme of features fusion for facial expression analysis: A facial action recognition,” Journal of Statistics and Management Systems, vol. 20, no. 4, pp. 693–701 (2017), doi: 10.1080/09720510.2017.1395189.[19] P. Sharma, M. S. P. Sharma, and R. S. Tomar, “A new approach for image segmentation using improved k-means and ROI saliency map,” Journal of Information and Optimization Sciences, vol. 38, no. 6, pp. 927–935 (2017), doi: 10.1080/02522667.2017.1372138. [20] L. J. B. Andrews and L. Raja, “A study on m-health in line with the sensors applying for a real time environment,” Journal of Statistics and Management Systems, vol. 20, no. 4, pp. 659–667 (2017), doi: 10.1080/09720510.2017.1395186.
Views: 49Downloads: 7Citations: 0
