The rapid emergence of generative models has given rise to hyper-realistic deepfakes and introduces threats to digital trust, privacy, and security. Traditional methods of detecting deepfake videos have relied heavily on either, hand-crafted visual features or deep convolutional networks (CNNs), but they cannot be generalized from one manipulation to another due to the distinct manipulation artifacts of deepfakes. This research investigates whether multimodal models such as contrastive language–image pre-training (CLIP), a model from OpenAI, can support deepfake detection in a robust way. The most distinct feature of CLIP is that it has been trained to accurately learn joint representations of natural language and images, which gives an obvious benefit in capturing semiotic mismatches that an analyst can visually overlook. By leveraging CLIP image embeddings and their associated textual prompt descriptors, the approach will assess whether there is an advantage to detecting deepfakes by understanding more nuanced artifacts in manipulated content. The effectiveness of this detection approach is assessed by evaluating linear classifiers for CLIP embeddings on benchmark deepfake datasets, while also investigating the performance of this method in relation to performance classifications across datasets. Our results highlighted that using textual context increases accuracy of detection, especially for cross-dataset and unseen manipulation scenarios. The approach also shows potential towards new forms of explainability by providing the user with interpretable reasoning behind the classification based on the language-driven prompt. Overall, we demonstrate that new forms of multimodal learning can effectively understand novel forms of multimedia-based forensics, and recognize CLIP’s potential as a base model for next-generation deepfake detection systems.