Mathematical modeling and AI-based AVOA-RSA algorithm for secure data communication

Sharma, Neha; Agarwal, Gaurav; Izhar, Jaweria

doi:10.47974/JDMSC-2530

Open Access Research Article

Mathematical modeling and AI-based AVOA-RSA algorithm for secure data communication

^*Neha SharmaCorresponding authornehav.sharma@jaipur.manipal.eduDepartment of Data Science and EngineeringVill Dehmi kalanManipal University JaipurJaipur, Rajasthan, 303007, IndiaView full profile → , Gaurav Agarwalgaggarwal@amity.uzDepartment of Information Technology and EngineeringAmity University in TashkentTashkent, 100128, UzbekistanView full profile → , Jaweria IzharJaweria.i@demont.ac.aeDepartment of ComputingWestford Education GroupDemont Institute of Management TechnologyDubai, 73000, U.A.E.View full profile →

* Corresponding author · click or hover a name for details

pp. 325–334Vol. 29Issue 1January 2026DOI: 10.47974/JDMSC-2530 Crossmark XML

Received:: 22 Apr 2025
Published Online:: 31 Dec 2025
Article type:: Research Article
Language:: EN
Article no.:: JDMSC-2530
Pages:: 325–334

Abstract

The increasing sophistication of cyber threats necessitates the development of more robust cryptographic systems. This paper introduces African Vulture Optimization-Enhanced RSA (AVO-RSA), a novel hybrid cryptosystem that integrates the African Vulture Optimization Algorithm (AVOA) with the traditional RSA algorithm to enhance security in data communication. AVO-RSA optimizes prime number selection and key generation using bio-inspired heuristics, making it more resistant to brute-force and factorization attacks. The proposed algorithm dynamically adjusts key parameters based on communication patterns, ensuring adaptive security. Experimental results demonstrate that AVO-RSA improves key strength while maintaining computational efficiency. This work contributes to the field of bio-inspired cryptography by introducing a nature-based optimization technique to strengthen asymmetric encryption. By optimizing prime selection using mathematical modelling and AI-driven search patterns, the proposed system significantly strengthens the resilience of RSA against modern cryptographic attacks. Experimental results demonstrate improvements in key unpredictability, encryption efficiency, and overall security, making it highly suitable for secure data communication.

Keywords

RSA African vulture optimization Cryptography Secure communication Bio-inspired algorithms

Subject Classifications

Primary 49M05Secondary 65K99

References

[1] R. L. Rivest, A. Shamir, and L. Adleman, “A method for obtaining digital signatures and public-key cryptosystems,” Commun. ACM, vol. 21, no. 2, pp. 120–126 (Feb. 1978).
[2] NIST, “Recommendation for key management: Part 1 – General,” *NIST Spec. Publ. 800-57*, Rev. 5 (2020).
[3] S. Mirjalili, “Genetic algorithms,” in Evolutionary Algorithms and Neural Networks, Springer, pp. 43–55 (2019).
[4] A. Kaveh and A. D. Eslamlou, “African vulture optimization: A novel nature-inspired metaheuristic algorithm,” Comput. Struct., vol. 252, 106559 (2021).
[5] X.-S. Yang, “Nature-inspired optimization algorithms: Challenges and open problems,” J. Comput. Sci., vol. 46, 101104 (2020).
[6] J. Jonsson and B. Kaliski, “Public-key cryptography standards (PKCS) #1: RSA cryptography specifications,” RFC 3447 (2003).
[7] M. Wiener, “Cryptanalysis of short RSA secret exponents,” IEEE Trans. Inf. Theory, vol. 36, no. 3, pp. 553–558 (May 1990).
[8] D. Boneh and G. Durfee, “Cryptanalysis of RSA with private key d less than N^0.292,” IEEE Trans. Inf. Theory, vol. 46, no. 4, pp. 1339–1349 (Jul. 2000).
[9] J. Hoffstein, , J. Pipher, and J. H. Silverman, “NTRU: A ring-based public key cryptosystem,” LNCS, vol. 1423, pp. 267–288 (1998).
[10] V. Lyubashevsky, L. Ducas, E. Kiltz, T. Lepoint, P. Schwabe, G. Seiler, and S. Bai, “CRYSTALS-Dilithium: Algorithm specifications and supporting documentation,” NIST PQC Stand. (2021).
[11] S. Kumar and D. Sharma, “Key generation in cryptography using elliptic-curve cryptography and genetic algorithm,” Engineering Proceedings, vol. 59, no. 1, p. 59 (2023).
[12] L. D. Singh and K. M. Singh, “Particle swarm optimization for secure RSA parameter selection,” J. Inf. Secur. Appl., vol. 58, 102734 (2021).
[11] N. Moustafa, J. Hu, and J. Slay, “A holistic review of Network Anomaly Detection Systems: A comprehensive survey,” J. Netw. Comput. Appl., vol. 128, pp. 33–55 (Feb. 2019), doi: 10.1016/j.jnca.2018.12.006.
[13] R. Yamashita, M. Nishio, R. K. G. Do, and K. Togashi, “Convolutional neural networks: an overview and application in radiology,” Insights Imaging, vol. 9, no. 4, Art. no. 4 (Aug. 2018), doi: 10.1007/s13244-018-0639-9.
[14] V. Ravi, S. Kp, and P. Poornachandran, “Evaluation of Recurrent Neural Network and its Variants for Intrusion Detection System (IDS),” Int. J. Inf. Syst. Model. Des., vol. 8, pp. 43–63 (Jul. 2017), doi: 10.4018/IJISMD.2017070103.
[15] N. Moustafa and J. Slay, “UNSW-NB15: a comprehensive data set for network intrusion detection systems (UNSW-NB15 network data set),” in 2015 Military Communications and Information Systems Conference (MilCIS), Nov. 2015, pp. 1–6. doi: 10.1109/MilCIS.2015.7348942.
[16] N. Sharma, N. S. Yadav, and S. Sharma, “Classification of UNSW-NB15 dataset using Exploratory Data Analysis using Ensemble Learning,” EAI Endorsed Trans. Ind. Netw. Intell. Syst., vol. 8, no. 29, Art. no. 29 (2021), doi: 10.4108/eai.13-10-2021.171319.
[17] Kumar, P. Dadheech, V. Singh, L. Raja, and R. C. Poonia, “An enhanced quantum key distribution protocol for security authentication,” Journal of Discrete Mathematical Sciences and Cryptography, vol. 22, no. 4, pp. 499–507 (May 2019), doi: https://doi.org/10.1080/09720529.2019.1637154.
[18] https://www.hyperelliptic.org/tanja/teaching/CS20/slides/20210104-kem-dem.pdf

Views: 196Downloads: 120Citations: 0