Deepfake detection using contrastive language–image pre-training (CLIP)
*Swati SahCorresponding authorswati.sah@sharda.ac.inDepartment of Computer Science & ApplicationsSharda School of Computing Science & Engineering (SSCSE)Sharda UniversityGreater Noida, Uttar Pradesh, 201310, IndiaView full profile → , Shivam Kumarshivamamar99@gmail.comDepartment of Computer Science & ApplicationsSharda School of Computing Science & Engineering (SSCSE)Sharda UniversityGreater Noida, Uttar Pradesh, 201310, IndiaView full profile → , Ravi Prakash Chaturvedirpchaturvedi51@gmail.comDepartment of Computer Science & TechnologyManav Rachna UniversityFaridabad, Haryana, 121004, IndiaView full profile → , Ganesh Kumar Dixitganesh.engineering@sanskaramuniversity.ac.inDepartment of Computer Science and EngineeringSchool of Engineering and TechnologySansakaram UniversityJhajjar, Haryana, 124108, IndiaView full profile → , Rejwan Bin Sulaimanrejwan.sulaiman@northumbria.ac.ukDepartment of Computer Science and TechnologyNorthumbria UniversityNewcastle-upon-Tyne, U.K.View full profile → , Saurabh Kumarsauraubhkumar26dec@gmail.comDepartment of Computer Science & ApplicationsSharda School of Computing Science & Engineering (SSCSE)Sharda UniversityGreater Noida, Uttar Pradesh, 201310, IndiaView full profile →
* Corresponding author · click or hover a name for details
- Received:
- 01 Dec 2025
- Published Online:
- 30 Sep 2026
- Article type:
- Research Article
- Language:
- EN
- Article no.:
- JIOS-2416
- Pages:
- 3207–3223
Abstract
The rapid emergence of generative models has given rise to hyper-realistic deepfakes and introduces threats to digital trust, privacy, and security. Traditional methods of detecting deepfake videos have relied heavily on either, hand-crafted visual features or deep convolutional networks (CNNs), but they cannot be generalized from one manipulation to another due to the distinct manipulation artifacts of deepfakes. This research investigates whether multimodal models such as contrastive language–image pre-training (CLIP), a model from OpenAI, can support deepfake detection in a robust way. The most distinct feature of CLIP is that it has been trained to accurately learn joint representations of natural language and images, which gives an obvious benefit in capturing semiotic mismatches that an analyst can visually overlook. By leveraging CLIP image embeddings and their associated textual prompt descriptors, the approach will assess whether there is an advantage to detecting deepfakes by understanding more nuanced artifacts in manipulated content. The effectiveness of this detection approach is assessed by evaluating linear classifiers for CLIP embeddings on benchmark deepfake datasets, while also investigating the performance of this method in relation to performance classifications across datasets. Our results highlighted that using textual context increases accuracy of detection, especially for cross-dataset and unseen manipulation scenarios. The approach also shows potential towards new forms of explainability by providing the user with interpretable reasoning behind the classification based on the language-driven prompt. Overall, we demonstrate that new forms of multimodal learning can effectively understand novel forms of multimedia-based forensics, and recognize CLIP’s potential as a base model for next-generation deepfake detection systems.
Keywords
Subject Classifications
References
[1] M. J. Malik, M. A. Khan, F. B. Hussain, and R. Sadiq, “DeepFake creation and detection: A survey,” IEEE Access, vol. 9, pp. 103574–103600 (2021).
[2] H. Heidari, A. Shafiei, S. Hashemi, and A. M. Rahmani, “A systematic review of deep learning techniques for deepfake detection,” IEEE Access, vol. 10, pp. 125614–125633 (2022).
[3] A. E. Abbas and A. Taeihagh, “Artificial intelligence and deepfakes: A new threat to society?,” AI & Society, vol. 37, no. 2, pp. 531–543 (2022).
[4] M. Smeu, Y. Zhang, H. Le, and L. Tran, “DeCLIP: Detection and localization of image manipulations with CLIP representations,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. Workshops, pp. 370–379 (2023).
[5] X. Tong, Y. Zhao, Z. Li, and J. Gao, “Exploring spatio-temporal inconsistencies for deepfake video detection with transformers,” in Proc. ACM Multimedia Conf., pp. 4561–4570 (2023).
[6] S. Wang, Y. Xu, J. Wu, and F. Li, “Knowledge-guided prompt learning for deepfake image detection,” in Proc. AAAI Conf. Artif. Intell., vol. 37, no. 2, pp. 2633–2641 (2023).
[7] S. Son, J. Lee, K. Min, and W. Kim, “Enhancing deepfake detection: Spatial-temporal preprocessing and self-attention ResI3D model,” in Proc. Int. Conf. Multimedia Comput. Commun. Technol., pp. 1–6 (2023).
[8] H. Lin, Y. Chen, X. Wang, and L. Zhang, “DeepFake detection with multi-scale convolution and vision transformer,” Digit. Signal Process., vol. 133, Art. no. 103895 (2023).
[9] Z. Wang, Z. Cheng, J. Xiong, X. Xu, T. Li, B. Veeravalli, and X. Yang, “A timely survey on vision transformer for deepfake detection,” arXiv preprint arXiv:2405.08463 (2024).
[10] J. Gu, Y. Xu, J. Sun, and W. Liu, “Exploiting deepfakes by analyzing temporal feature inconsistency,” Int. J. Adv. Comput. Sci. Appl., vol. 14, no. 12, pp. 908–915 (2023).
[11] Y. Huang, F. Juefei-Xu, Q. Guo, Y. Liu, and G. Pu, “FakeLocator: Robust localization of GAN-based face manipulations,” IEEE Trans. Inf. Forensics Security, vol. 17, pp. 2657–2672 (2022).
[12] Y. Ni, D. Meng, C. Yu, C. Quan, D. Ren, and Y. Zhao, “CORE: COnsistent REpresentation learning for face forgery detection,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. Workshops, pp. 12–21 (2022).
[13] S. A. Khan and D. T. Dang-Nguyen, “Hybrid transformer network for deepfake detection,” in Proc. Int. Conf. Content-Based Multimedia Indexing, pp. 1–6 (2022).
[14] Z. Gu, Y. Chen, T. Yao, S. Ding, J. Li, F. Huang, and L. Ma, “Region-aware temporal inconsistency learning for deepfake video detection,” in Proc. Int. Joint Conf. Artif. Intell. (IJCAI), pp. 920–926 (2022).
[15] Y. Mirsky and W. Lee, “The creation and detection of deepfakes: A survey,” ACM Computing Surveys, vol. 54, no. 1, pp. 1–41 (Apr. 2021).
[16] Z. Khanjani, G. Watson, and V. P. Janeja, “How deep are the fakes? Focusing on audio deepfake: A survey,” arXiv preprint arXiv:2111.14203 (Nov. 2021).
[17] P. Yu, Z. Xia, J. Fei, and Y. Lu, “A survey on deepfake video detection,” IET Biometrics, vol. 10, no. 6, pp. 607–624 (Dec. 2021).
[18] L. Verdoliva, “Media forensics and deepfakes: An overview,” IEEE Journal of Selected Topics in Signal Processing, vol. 14, no. 5, pp. 910–932 (Aug. 2020).
[19] R. Tolosana, R. Vera-Rodriguez, J. Fierrez, A. Morales, and J. Ortega-Garcia, “Deepfakes and beyond: A survey of face manipulation and fake detection,” Information Fusion, vol. 64, pp. 131–148 (Dec. 2020).
[20] J. Zhang, Z. Wang, and H. Liu, “A survey on face forgery detection of deepfake,” in Proc. Int. Conf. Digital Image Processing (ICDIP), pp. 1–7 (Jun. 2020).
[21] P. Mounika, K. V. S. Reddy, and N. Ramakrishnaiah, “Comprehensive review and analysis on multi modal image retrieval,” J. Inf. Optim. Sci., vol. 46, no. 2, pp. 403–413 (2025), doi: 10.47974/JIOS-1923.
[22] C. Leelavathi, G. Thahniyath, and K. Rajendra Prasad, “Optimizing the deep features using the spectral concept for an efficient video classification,” J. Inf. Optim. Sci., vol. 46, no. 2, pp. 553–562 (2025), doi: 10.47974/JIOS-1959.
