Audio deepfake detection (ADD) models are critical for countering the malicious use of text-to-speech (TTS) models. Evaluating and strengthening ADD models requires developing datasets that span the space of generated audio and highlight high-error regions. Existing dataset development strategies face two challenges: (i) manual collection, and (ii) inefficient discovery of blind spots in the ADD models. To address these challenges, we propose FoeGlass, the first black-box automated red-teaming method for ADDs, which effectively discovers ADD failure modes in the space of generated audio underexplored by state-of-the-art deepfake benchmarks. FoeGlass uses the in-context learning capabilities of an LLM to explore the input space of a TTS model, generating audio samples that fool the target ADD using only black-box access to all components. By using a carefully designed context based on diversity measurements, FoeGlass mitigates the common problem of mode collapse in automated red-teaming systems. Empirical evaluations on several open-source ADD and TTS models demonstrate that data generated from FoeGlass substantially improves the false negative rates over unconditional sampling baselines and recent spoofing datasets by up to 94%, while requiring no manual supervision. Furthermore, we show that the attacks generated by FoeGlass are transferable across different target ADDs, demonstrating its broad applicability and ease of use for the automated red teaming of ADD systems. Finally, fine-tuning ADD models on FoeGlass-generated samples notably enhances the robustness of the detectors (up 41%).

翻译：音频深度伪造检测（ADD）模型对于应对文本转语音（TTS）模型的恶意使用至关重要。评估和增强ADD模型需要开发涵盖生成音频空间并突出高错误区域的数据集。现有数据集开发策略面临两个挑战：（i）人工收集，（ii）低效发现ADD模型盲区。为应对这些挑战，我们提出FoeGlass——首个针对ADD的黑盒自动化红队测试方法，其能有效发现被最先进深度伪造基准所忽视的生成音频空间中ADD失效模式。FoeGlass利用大语言模型（LLM）的上下文学习能力探索TTS模型的输入空间，仅通过黑盒访问所有组件即可生成欺骗目标ADD的音频样本。通过基于多样性度量精心设计的上下文，FoeGlass缓解了自动化红队测试系统中常见的模式坍塌问题。在多个开源ADD和TTS模型上的实证评估表明，与无条件采样基线及近期欺骗性数据集相比，FoeGlass生成的数据能显著将假阴性率提升高达94%，且无需人工监督。此外，我们证明FoeGlass生成的攻击可跨不同目标ADD迁移，展示了其在ADD系统自动化红队测试中的广泛适用性和易用性。最后，在FoeGlass生成的样本上微调ADD模型能显著增强检测器的鲁棒性（提升高达41%）。