Discrete graph spectral framework for dynamic malware detection in software-defined networks
*S. SivaprakashCorresponding authorsivaprakash.s@vit.ac.inSchool of Computer Science and Engineering (SCOPE)Vellore Institute of TechnologyVellore, Tamil Nadu, 632014, IndiaView full profile → , S. Dilip Kumardilipkumar.s@vit.ac.inSchool of Computer Science and Engineering (SCOPE)Vellore Institute of TechnologyVellore, Tamil Nadu, 632014, IndiaView full profile → , S. Thamizharasanthamizharasan.s@vit.ac.inSchool of Computer Science and Engineering (SCOPE)Vellore Institute of TechnologyVellore, Tamil Nadu, 632014, IndiaView full profile → , P. Iyappaniyappan.perumal@vit.ac.inSchool of Computer Science and Engineering (SCOPE)Vellore Institute of TechnologyVellore, Tamil Nadu, 632014, IndiaView full profile → , W. T. Chembianwtchembian@velhightech.comDepartment of Computer Science and EngineeringVel Tech High Tech Dr. Rangarajan Dr. Sakunthala Engineering College (Autonomous)Chennai, Tamil Nadu, 600062, IndiaView full profile → , K. Sonysonykarra@kluniversity.inDepartment of Electronics & Communication EngineeringKoneru Lakshmaiah Education FoundationVaddeswaram, Andhra Pradesh, 522502, IndiaView full profile →
* Corresponding author · click or hover a name for details
- Received:
- 01 Nov 2025
- Published Online:
- 30 Sep 2026
- Article type:
- Research Article
- Language:
- EN
- Article no.:
- JDMSC-2810
- Pages:
- 3693–3702
Abstract
This paper proposes a novel Graph Neural Network-based Dynamic Malware (GNNDM) detection system that models network traffic and system call sequences as time-evolving attributed graphs, enabling the capture of complex relational dependencies that scalar feature vectors cannot represent. The proposed architecture employs discrete graph spectral analysis, leveraging the eigen spectrum of the normalized graph Laplacian matrix to construct permutation-invariant node embeddings that characterize both local topological neighbourhoods and global network communication patterns. An edge attention mechanism weighted by temporal co-occurrence statistics and entropy-differential features enhances the model’s sensitivity to subtle behavioural deviations indicative of covert lateral movement and command-and-control (C2) communication. The GNN-DM model is trained on a hybrid dataset comprising NSL-KDD, CICIDS-2021, and a proprietary SDN traffic trace containing 42 malware families collected in a high-fidelity testbed. Experimental results demonstrate that GNN-DM achieves 98.4% detection accuracy, 97.9% F1-score, and a false positive rate of 0.9%, outperforming CNN-LSTM baseline by 4.2 percentage points and SVM-based classifiers by 6.7 percentage points.
Keywords
Subject Classifications
References
[1] X. Zhou, W. Liang, K. I.-K. Wang, H. Wang, Q. Jin, and J. J. P. C. Rodrigues, “Deep learning enhanced edge intelligence for intrusion detection in software defined networks,” IEEE Trans. Netw. Service Manag., vol. 18, no. 3, pp. 3597–3608 (Sep. 2021).
[2] T. Bilot, N. El Madhoun, K. Al Agha, and A. Zouaoui, “Graph neural networks for intrusion detection: A survey,” IEEE Access, vol. 10, pp. 38780–38816 (2022).
[3] W. W. Lo, S. Layeghy, M. Sarhan, M. Gallagher, and M. Portmann, “E-GraphSAGE: A graph neural network based intrusion detection system for IoT,” in Proc. IEEE/IFIP Netw. Oper. Manage. Symp. (NOMS), pp. 1–9 (2022).
[4] A. Narayanan, C. Shi, and B. J. Bhowmick, “CTDGNN: Continuous-time dynamic graph neural networks for network intrusion event detection,” IEEE Trans. Inf. Forensics Security, vol. 18, pp. 1490–1503 (2023).
[5] Y. Peng, G. Jiang, T. Claypool, and F. Xu, “Laplacian spectrum analysis for encrypted malicious traffic classification in SDN environments,” Comput. Commun., vol. 183, pp. 11–22 (Jan. 2022).
[6] X. Chen and R. Wang, “Attention-based graph neural network for malware call graph classification,” in Proc. IEEE Int. Conf. Data Sci. Cybersecurity (DSC), pp. 1–8 (2022).
[7] D. Vasan, M. Alazab, S. Wassan, H. Naeem, B. Safaei, and Q. Zheng, “IMCFN: Image-based malware classification using fine-tuned convolutional neural network architecture,” Comput. Netw., vol. 171, Art. no. 107138 (Apr. 2020).
[8] E. Caville, W. W. Lo, S. Layeghy, and M. Portmann, “Anomal-E: A self-supervised network intrusion detection system based on graph neural networks,” Knowl.-Based Syst., vol. 258, Art. no. 110030 (2022).
[9] Y. Yang, Q. Xiong, Y. Huang, and K.-K. R. Choo, “Adversarial robustness of GNN-based intrusion detection systems against graph injection attacks,” IEEE Trans. Dependable Secure Comput., vol. 20, no. 6, pp. 5157–5170 (2023).
[10] M. Sarhan, S. Layeghy, N. Moustafa, and M. Portmann, “NetFlow datasets for machine learning-based network intrusion detection systems: A systematic review,” in Proc. ACM Int. Conf. Big Data Comput., Appl. Technol. (BDCAT), pp. 9–17 (2022).
[11] S. Salunkhe, A. O. Mulani, D. Shahane, M. Rana, S. M. Shukla, and M. M. Jadhav, “Secure image transmission using chaotic encryption and DWT watermarking on reconfigurable platform,” J. Discrete Math. Sci. Cryptogr., vol. 29, no. 1, pp. 349–361 (2026).
[12] K. R. Chaudhari, A. O. Mulani, M. P. Gajare, V. Jadhav, P. Yawle, and A. V. Bang, “Bit error rate analysis of various error correction codes with concatenated RS-convolutional codes,” J. Discrete Math. Sci. Cryptogr., vol. 29, no. 1, pp. 199–214 (2026).
