TARU PUBLICATIONS
Journal of Information and Optimization Sciences cover

Journal of Information and Optimization Sciences

(Since 1980)
Hybrid ·Peer-reviewed·ISSN (Online): 2169-0103·ISSN (Print): 0252-2667

WoS  JIF 2026 : 0.4 (Q4)

Powered by:Powered by

Monthly Journal: Publishes theoretical and applied research on topics in information and optimization sciences.

Issues up to 2022 co-published with and available at:Taylor & Francis
submissions@tarupublications.com
Taru Online First Current IssueSpecial IssuesBrowse all Issues
Journal of Information and Optimization Sciences
Open Access Research Article

Exploring alternative techniques for accelerating deep learning convergence for batch normalization

, * , , , ,

* Corresponding author · click or hover a name for details

pp. 1165–1175Vol. 46Issue 4-BMay 2025DOI: 10.47974/JIOS-1900XML
Received:
16 Oct 2024
Published Online:
31 May 2025
Article type:
Research Article
Language:
EN
Article no.:
JIOS-1900
Pages:
1165–1175

Abstract

Deep learning models are the most important part of current AI systems because they can learn complex patterns from data in a very impressive way. However, creating these models often takes a lot of time and computer power, especially when methods like Batch Normalization (BN) are used. By balancing activations, BN helps to stabilize and speed up neural network training, but it adds extra work to the computer, which slows down convergence speed. This paper looks at different ways to speed up the convergence of deep learning models that use BN. This study looks into different ways to slow down the convergence process caused by BN layers by reading a lot of current research and doing experiments. Some methods, like weight normalization, group normalization, and layer normalization, are looked at to see if they can be used instead of or in addition to BN to keep or speed up convergence. It also looked into how to change the architecture and make optimization techniques that work with these different standardization methods. Our results show that while BN is still a popular way to keep neural network training stable, other methods show promise for speeding up convergence without affecting performance. By using these other approaches, professionals can cut down on the amount of work that needs to be done on computers and the time it takes to train models. This makes it easier to build and use deep learning models in places with limited resources. This study adds to the ongoing search for deep learning methods that work well and can be used by many people. It also opens up new areas for research and improvement in model training.

Keywords

Deep learningBatch normalizationConvergence accelerationWeight normalizationGroup normalizationLayer normalization

Subject Classifications

68Q25

References

[1] S. Wu, G. Li, F. Chen, and L. Shi, “L1 -Norm Batch Normalization for Efficient Training of Deep Neural Networks,” in IEEE Transactions on Neural Networks and Learning Systems, vol. 30, no. 7, pp. 2043-2051 (July 2019).
[2] Y. S. Ting, Y. F. Teng and T. D. Chiueh, “Batch Normalization Processor Design for Convolution Neural Network Training and Inference,” 2021 IEEE International Symposium on Circuits and Systems (ISCAS), Daegu, Korea, pp. 1-4 (2021)
[3] A. Muhammad, F. Shamshad and S. H. Bae, “Adversarial Attacks and Batch Normalization: A Batch Statistics Perspective,” in IEEE Access, vol. 11, pp. 96449-96459 (2023).
[4] M. Alwani, H. Chen, M. Ferdman, and P. Milder, “Fused-layer CNN accelerators,” in Proceedings of the 49th Annual IEEE/ACM International Symposium on Microarchitecture (MICRO), pp. 1–12 (2016), doi: 10.1109/MICRO.2016.7783722.
[5] Z. Yang, B. Ni, X. Huang, and D. Chen, “Bactran: A hardware batch normalization implementation for CNN training engine,” IEEE Embedded Systems Letters, vol. 12, no. 4, pp. 111–114 (Dec. 2020), doi: 10.1109/LES.2020.3002759.
[6] S. Du, J. Liu and L. Zhao, “Research on Rolling Bearing Condition Monitoring Method Based on Deep Learning,” 2020 Chinese Automation Congress (CAC), Shanghai, China, pp. 74-78 (2020).
[7] Y. Li, T. Zhang, and C. Shan, Deep Learning Convolutional Neural Network: From Entry to Mastery. Beijing, China: Mechanical Industry Press (2018).
[8] W. F. Hidayat, M. F. Julianto, Y. Malau, A. Setiadi and Sriyadi, “Implementation of LSTM and Adam Optimization as a Cryptocurrency Polygon Price Predictor,” 2023 International Conference on Information Technology Research and Innovation (ICITRI), Jakarta, Indonesia, pp. 123-127 (2023).
[9] G. Ardiyansyah, F. Ferdiansyah and U. Ependi, “Deep Learning Model Analysis and Web-Based Implementation of Cryptocurrency Prediction”, J. Inf. Syst. Informatics, vol. 4, no. 4, pp. 958-974 (2022).
[10] K. Maharana, S. Mondal and B. Nemade, “A review: Data pre-processing and data augmentation techniques”, Glob. Transitions Proc., vol. 3, no. 1, pp. 91-99 (2022).
[11] S. Saika, S. Shewali, M. J. Borah, and D. J. Mahanta, “Arithmetic mean of interval valued intuitionistic fuzzy soft sets and its application in diagnosing infectious diseases,” J. Interdiscip. Math., vol. 27, no. 5, pp. 1079–1090 (2024), doi: 10.47974/JIM-1934.
[12] P. Benz, C. Zhang and I. S. Kweon, “Batch normalization increases adversarial vulnerability and decreases adversarial transferability: A non-robust feature perspective”, Proc. IEEE/CVF Int. Conf. Comput. Vis. (ICCV), pp. 7798-7807 (Oct. 2021).
[13] J. Bruce and D. Erman, “A probabilistic approach to systems of parameters and noether normalization”, arXiv:1604.01704 (2016).

Views: 230Downloads: 80Citations: 0