The rapid rise of deepfake technology poses a severe threat to social and political stability by enabling hyper-realistic synthetic media capable of manipulating public perception. However, existing detection methods struggle with two core limitations: (1) modality fragmentation, which leads to poor generalization across diverse and adversarial deepfake modalities; and (2) shallow inter-modal reasoning, resulting in limited detection of fine-grained semantic inconsistencies. To address these, we propose ConLLM (Contrastive Learning with Large Language Models), a hybrid framework for robust multimodal deepfake detection. ConLLM employs a two-stage architecture: stage 1 uses Pre-Trained Models (PTMs) to extract modality-specific embeddings; stage 2 aligns these embeddings via contrastive learning to mitigate modality fragmentation, and refines them using LLM-based reasoning to address shallow inter-modal reasoning by capturing semantic inconsistencies. ConLLM demonstrates strong performance across audio, video, and audio-visual modalities. It reduces audio deepfake EER by up to 50%, improves video accuracy by up to 8%, and achieves approximately 9% accuracy gains in audio-visual tasks. Ablation studies confirm that PTM-based embeddings contribute 9%-10% consistent improvements across modalities.

翻译：深度伪造技术的迅速兴起对社会和政治稳定构成了严重威胁，其生成的超逼真合成媒体能够操纵公众认知。然而，现有检测方法面临两个核心局限：(1) 模态碎片化，导致对不同类型及对抗性深度伪造模态的泛化能力较差；(2) 浅层跨模态推理，导致对细粒度语义不一致性的检测能力有限。为解决这些问题，我们提出ConLLM（基于大语言模型的对比学习框架），一种用于鲁棒多模态深度伪造检测的混合框架。ConLLM采用两阶段架构：第一阶段使用预训练模型提取模态特定嵌入；第二阶段通过对比学习对齐这些嵌入以缓解模态碎片化，并利用基于大语言模型的推理进行精炼，通过捕捉语义不一致性来解决浅层跨模态推理问题。ConLLM在音频、视频及音视频模态上均表现出强大性能：将音频深度伪造的等错误率降低最高达50%，视频检测准确率提升最高达8%，在音视频任务中实现约9%的准确率增益。消融实验证实，基于预训练模型的嵌入为各模态带来了9%-10%的稳定性能提升。