The prevalence and strong capability of large language models (LLMs) present significant safety and ethical risks if exploited by malicious users. To prevent the potentially deceptive usage of LLMs, recent works have proposed algorithms to detect LLM-generated text and protect LLMs. In this paper, we investigate the robustness and reliability of these LLM detectors under adversarial attacks. We study two types of attack strategies: 1) replacing certain words in an LLM's output with their synonyms given the context; 2) automatically searching for an instructional prompt to alter the writing style of the generation. In both strategies, we leverage an auxiliary LLM to generate the word replacements or the instructional prompt. Different from previous works, we consider a challenging setting where the auxiliary LLM can also be protected by a detector. Experiments reveal that our attacks effectively compromise the performance of all detectors in the study with plausible generations, underscoring the urgent need to improve the robustness of LLM-generated text detection systems.

翻译：大型语言模型（LLMs）的广泛应用和强大能力在恶意用户利用时会带来显著的安全与伦理风险。为防止LLMs的潜在欺骗性使用，近期研究提出了检测LLM生成文本并保护LLMs的算法。本文研究了这些LLM检测器在对抗攻击下的鲁棒性和可靠性。我们分析了两类攻击策略：1）在给定上下文中将LLM输出中的某些词语替换为同义词；2）自动搜索指令提示以改变生成的写作风格。两种策略均借助辅助LLM生成词语替换或指令提示。与以往研究不同，我们考虑了一个更具挑战性的设置：辅助LLM本身也可能受到检测器保护。实验表明，我们的攻击能通过合理的生成内容有效削弱研究中所有检测器的性能，凸显了提升LLM生成文本检测系统鲁棒性的迫切需求。