This paper proposes a novel deep learning approach for real-time activity classification in video surveillance applications. In this paper, we put forward a light mobile network-based LSTM called MobileSTM and an Inception V3 Network and combine their advantages through ensemble methods. Lightweight in architecture, the MobileNet can guarantee efficient processing on resource-constrained devices. LSTMs are powerful models in mining temporal information within sequences. Inception V3, known for its deep architecture and high accuracy, further enhances the model’s capability to classify activities. MobileSTM and Inception V3 networks are combined and trained to recognize and classify these activities into pre-defined vital categories for public safety: violent acts and normal activities. The following approach brings forth the idea of strengthening the real-time video analysis for security personnel, so that they could focus their efforts and respond promptly in cases when some threats are observed or indicated.