The proliferation of large language models has raised growing concerns about their misuse, particularly in cases where AI-generated text is falsely attributed to human authors. Machine-generated content detectors claim to effectively identify such text under various conditions and from any language model. This paper critically evaluates these claims by assessing several popular detectors (RADAR, Wild, T5Sentinel, Fast-DetectGPT, GPTID, LogRank, Binoculars) on a range of domains, datasets, and models that these detectors have not previously encountered. We employ various prompting strategies to simulate adversarial attacks, demonstrating that even moderate efforts can significantly evade detection. We emphasize the importance of the true positive rate at a specific false positive rate (TPR@FPR) metric and demonstrate that these detectors perform poorly in certain settings, with [email protected] as low as 0%. Our findings suggest that both trained and zero-shot detectors struggle to maintain high sensitivity while achieving a reasonable true positive rate.

翻译：随着大型语言模型的广泛使用，其潜在滥用问题日益引发关注，尤其是在人工智能生成文本被错误归因于人类作者的情形下。机器生成内容检测器声称能够在各种条件下有效识别来自任意语言模型的此类文本。本文通过评估多款主流检测器（RADAR、Wild、T5Sentinel、Fast-DetectGPT、GPTID、LogRank、Binoculars）在检测器未曾接触过的领域、数据集和模型上的表现，对这些声明进行了批判性检验。我们采用多种提示策略模拟对抗性攻击，证明即使中等程度的规避努力也能显著逃逸检测。我们重点分析了特定误报率下的真阳性率（TPR@FPR）指标的重要性，并证明这些检测器在特定场景下性能欠佳，其[email protected]指标可低至0%。研究结果表明，无论是经过训练的检测器还是零样本检测器，都难以在保持高灵敏度的同时获得合理的真阳性率。