In an age of sophisticated cyber threats, secure digital communications are a necessity. Conventional cryptographic protocols like AES and RSA offer data confidentiality and integrity, but they are static protocols that have no way of responding to adaptive threats. In this paper, a deep learning-based intrusion detection system and hybrid AES-RSA security enhancement is proposed driven by an AI. Using UNC-W-NB15 dataset, an LSTM network is trained to recognize malicious traffic patterns, and the trained LSTM can work in real time to detect intrusions or anomalies in encrypted communication. In this proposed protocol, AES is used for high speed data encryption, RSA for secure key exchange, and along with the alerts of LSTM, dynamically adjust or reinforce the encryption parameters as threats are encountered. Experimental results indicate that this integrated approach leads to huge security outcomes gain, 97.8% intrusion detection accuracy, 95% attack detection and so on—far better than using encryption only conventionally. In addition, with the use of an AI driven cryptographic protocol, we achieve strong performance metrics, such as higher throughput (up to 150Mbps), lower decryption latency, meaning strong security can be made without a degradation in efficiency. These results demonstrate that encryption protocols can achieve near ideal security against a plaintext access attack, and when combined with machine intelligence, these protocols can be joined with practical digital communication systems without sacrificing any performance.