TARU PUBLICATIONS
Journal of Discrete Mathematical Sciences and Cryptography cover
Open Access ·Peer-reviewed·ISSN (Online): 2169-0065·ISSN (Print): 0972-0529

Monthly Journal: Publishes theoretical and applied research in all areas of Discrete Mathematical Sciences, Cryptography, Combinatorics, Elliptic Curves and Information Security.

Issues up to 2022 co-published with and available at:Taylor & Francis Online
submissions@tarupublications.com
Open Access Research Article

Network-based anomaly detection in encrypted data streams : A cryptanalysis perspective

* , , , , ,

* Corresponding author · click or hover a name for details

pp. 2115–2124Vol. 28Issue 5-BAugust 2025DOI: 10.47974/JDMSC-2428 Crossmark XML
Received:
05 Nov 2024
Published Online:
02 Sep 2025
Article type:
Research Article
Language:
EN
Article no.:
JDMSC-2428
Pages:
2115–2124

Abstract

Encrypted traffic now comprises most Internet flows, rendering traditional inspection methods ineffective and posing significant challenges for real-time anomaly detection. In this work, we introduce a layered detection framework that combines (1) a cryptanalysis-informed scoring function quantifying deviations in flow entropy and handshake parameter statistics with (2) a hybrid decision pipeline that employs a lightweight decision tree to filter benign traffic before invoking a secondary SVM/Random-Forest classifier, and (3) an adaptive thresholding and retraining strategy based on exponentially weighted moving averages and daily model updates. Evaluated on real-world TLS 1.3 captures, the CTU-13 botnet dataset, and instrumented QUIC traces, our approach achieves a 92 % detection rate and a 2 % false-positive rate, while processing each flow in under 1 ms. These results demonstrate that fusing domain-specific cryptanalytic insights with adaptive machine learning yields both high accuracy and operational efficiency for encrypted-traffic monitoring.

Keywords

Subject Classifications

68Q11

References

[1] S. Yu and Y. Won, “A survey of methods for encrypted network traffic fingerprinting,” Mathematical Biosciences and Engineering, vol. 20, no. 2, pp. 2183–2202 (Nov. 2022), doi:10.3934/mbe.2023101.
[2] G. Long and Z. Zhang, “Deep Encrypted Traffic Detection: An Anomaly Detection Framework for Encryption Traffic Based on Parallel Automatic Feature Extraction,” Computational Intelligence and Neuroscience, vol. 2023, Art. ID 3316642, 12 pp. (Mar. 2023), doi:10.1155/2023/3316642. 
[3] H. Yu, W. Yang, B. Cui, R. Sui, and X. Wu, “Renyi entropy-driven network traffic anomaly detection with dynamic threshold,” Cybersecurity, vol. 7, Art. no. 64 (Dec. 2024), doi:10.1186/s42400-024-00249-1.
[4] J. Zhao, Q. Li, and Z. Han, “ReTrial: Robust Encrypted Malicious Traffic Detection via Discriminative Relation Incorporation and Misleading Relation Correction,” IEEE Transactions on Information Forensics and Security, early access (Jan. 2024), doi:10.1109/TIFS.2024.3515821.
[5] O.-A. Ticleanu, T. Popa, D. I. Hunyadi, and N. Constantinescu, “Detecting Encrypted and Unencrypted Network Data Using Entropy Analysis and Confidence Intervals,” Entropy, vol. 25, no. 3, Art. 397 (Feb. 2023), doi:10.3390/e25030397.
[6] I. A. Alwhbi, C. C. Zou, and R. N. Alharbi, “Encrypted Network Traffic Analysis and Classification Utilizing Machine Learning,” Sensors, vol. 24, no. 11, Art. 3509 (Jun. 2024), doi:10.3390/s24113509.
[7] M. Phatak and M. Patwardhan, “Mathematical modelling and statistical analysis in designing deep learning-based shot boundary detection and aesthetic assessment of videos,” J. Interdiscip. Math., vol. 27, no. 2, pp. 369–382 (2024), doi: 10.47974/JIM-1857.
[8] H. A. A. Mohammed, Z. S. Ktran, and B. S. M. Ali, “Feasible and optimal solutions for linear programming in modulo n space,” J. Interdiscip. Math., vol. 28, no. 3-A, pp. 755–762 (2025), doi: 10.47974/JIM-1970.

Views: 953Downloads: 17Citations: 2