Modern digital infrastructures (such as cloud computing, IoT systems and distributed applications) heavily rely on secure communication protocols in which confidentiality, integrity, and authentication are essential. Nevertheless, the growing complexity of such systems and the development of advanced cyber-attacks reveal serious weaknesses of current methods of verification, especially their scalability and computation costs, as well as the possibility to simulate adversarial behavior. In this paper, the author discusses these issues with a proposal of a finite automata-based system of verifying a secure protocol. This work is mainly aimed at creating a highly mathematical and computationally efficient model of modeling and verifying communication protocols in adversarial conditions. In the proposed methodology, the protocols are represented as finite automata in which states are the configuration of the system, and transitions are the exchange of messages between the states. Adversary model is composed with automata and security requirements like confidentiality, integrity and authentication are formalized as constraints. To verify that the violations occur, the analysis of reachability and language-theoretic methods are used to identify the violations by exploring unsafe states. The findings indicate that the proposed framework is highly scalable and highly accurate with moderate computational cost which is better than conventional approaches to model checking, Petri nets and logic-based approaches. The measures of evaluation are the complexity of the state, verification time, overhead of transition, and memory usage, which demonstrate a better efficiency and effective attack detection possibilities.