In the age of increasingly realistic generative AI, robust deepfake detection is essential for mitigating fraud and disinformation. While many deepfake detectors report high accuracy on academic datasets, we show that these academic benchmarks are out of date and not representative of real-world deepfakes. We introduce Deepfake-Eval-2024, a new deepfake detection benchmark consisting of in-the-wild deepfakes collected from social media and deepfake detection platform users in 2024. Deepfake-Eval-2024 consists of 45 hours of videos, 56.5 hours of audio, and 1,975 images, encompassing the latest manipulation technologies. The benchmark contains diverse media content from 88 different websites in 52 different languages. We find that the performance of open-source state-of-the-art deepfake detection models drops precipitously when evaluated on Deepfake-Eval-2024, with AUC decreasing by 50% for video, 48% for audio, and 45% for image models compared to previous benchmarks. We also evaluate commercial deepfake detection models and models finetuned on Deepfake-Eval-2024, and find that they have superior performance to off-the-shelf open-source models, but do not yet reach the accuracy of deepfake forensic analysts. The dataset is available at https://github.com/nuriachandra/Deepfake-Eval-2024.

翻译：在生成式人工智能日益逼真的时代，稳健的深度伪造检测对于减少欺诈和虚假信息至关重要。尽管许多深度伪造检测器在学术数据集上报告了高准确率，但我们表明这些学术基准已过时，且不能代表真实世界的深度伪造。我们提出了Deepfake-Eval-2024，这是一个新的深度伪造检测基准，由2024年从社交媒体和深度伪造检测平台用户收集的野外深度伪造组成。Deepfake-Eval-2024包含45小时视频、56.5小时音频和1975张图像，涵盖了最新的篡改技术。该基准包含来自52种不同语言的88个不同网站的多样化媒体内容。我们发现，在Deepfake-Eval-2024上评估时，开源最先进深度伪造检测模型的性能急剧下降，其中视频模型的AUC相比先前基准下降50%，音频模型下降48%，图像模型下降45%。我们还评估了商业深度伪造检测模型和基于Deepfake-Eval-2024微调的模型，发现它们优于现成的开源模型，但尚未达到深度伪造取证分析师的准确率。数据集可在https://github.com/nuriachandra/Deepfake-Eval-2024获取。