Recent network environments are exposed to expanding threats that question the integrity of the conventional cryptographic processes. Conventional systems including RSA and ECC are effective but show the limitation in providing a balance between computational efficiency and robust security, particularly, with regard to changing attack models. This research highlights a missing link in incorporating structured mathematical frameworks, specifically discrete mathematics, for the protocol improvement. To overcome this, a new network security protocol is proposed by integrating the Finite Field Arithmetic with Prime Graph Structures. Finite fields provide secure key generation and encryption via modular operations and prime graphs provide trust-based topology because communication is possible only between co-prime connected nodes. The methodology provides the cryptographic strength with the topological robustness. Experimental evaluation shows that the proposed model delivers the reduced encryption time (15.2 ms), increased key exchange success rate (97.6%) and better attack resistance (96.3%) as against the standard RSA and ECC protocols. Also, the system has less computational overhead and is therefore appropriate for real-time applications.