This paper presents a discrete mathematical framework for machine learning-based cryptanalysis aimed at the automatic identification of classical encryption algorithms. Traditional cryptanalysis methods rely heavily on manual statistical analysis, whereas recent machine learning approaches often lack theoretical justification for feature separability. To address this limitation, this study introduces a novel hybrid statistical invariant termed the Normalized Cipher Randomness Index (NCRI), which integrates Shannon entropy and the Index of Coincidence into a unified mathematical formulation. The proposed framework extracts statistical features from ciphertext, including entropy, Index of Coincidence, repetition rate, and letter frequency distribution, and augments them with NCRI to enhance discriminative capability. A theoretical analysis is developed to establish that NCRI provides separability between monoalphabetic and polyalphabetic ciphers. The extracted features are used to train multiple supervised learning models, including Random Forest, Support Vector Machine (SVM), and XGBoost, with an ensemble learning strategy employed to improve robustness. Experimental evaluation demonstrates that the proposed approach achieves high classification accuracy of approximately 99%, with clear statistical separation between cipher types. The results confirm that NCRI significantly enhances feature representation and improves classification performance. The proposed framework provides a theoretically grounded and computationally efficient solution for automated cryptanalysis and highlights the potential of integrating discrete mathematical modeling with machine learning techniques in cryptographic security analysis.