Large language models (LLMs) have shown remarkable performance in various tasks and have been extensively utilized by the public. However, the increasing concerns regarding the misuse of LLMs, such as plagiarism and spamming, have led to the development of multiple detectors, including fine-tuned classifiers and statistical methods. In this study, we equip LLMs with prompts, rather than relying on an external paraphraser, to evaluate the vulnerability of these detectors. We propose a novel Substitution-based In-Context example Optimization method (SICO) to automatically construct prompts for evading the detectors. SICO is cost-efficient as it requires only 40 human-written examples and a limited number of LLM inferences to generate a prompt. Moreover, once a task-specific prompt has been constructed, it can be universally used against a wide range of detectors. Extensive experiments across three real-world tasks demonstrate that SICO significantly outperforms the paraphraser baselines and enables GPT-3.5 to successfully evade six detectors, decreasing their AUC by 0.5 on average. Furthermore, a comprehensive human evaluation show that the SICO-generated text achieves human-level readability and task completion rates, while preserving high imperceptibility. Finally, we propose an ensemble approach to enhance the robustness of detectors against SICO attack. The code is publicly available at https://github.com/ColinLu50/Evade-GPT-Detector.

翻译：大型语言模型（LLMs）在各类任务中展现出卓越性能，并已被公众广泛使用。然而，针对LLMs滥用问题（如剽窃和垃圾信息）的日益担忧，促使多种检测器（包括微调分类器与统计方法）得以研发。本研究通过为LLMs配置提示词（而非依赖外部改写器），评估这些检测器的脆弱性。我们提出一种新颖的基于替换的上下文示例优化方法（SICO），可自动构建用于规避检测的提示词。SICO具有成本效益——仅需40个人工撰写的示例及有限次LLM推理即可生成提示词。此外，一旦构建出任务特定提示词，便可通用性地对抗多种检测器。在三个真实任务中的大量实验表明，SICO显著优于基于改写器的基线方法，使GPT-3.5成功规避六种检测器，平均降低其AUC值0.5。同时，全面的人工评估显示，SICO生成的文本既达到人类水平的可读性与任务完成率，又保持了高度隐蔽性。最后，我们提出一种集成方法以增强检测器对SICO攻击的鲁棒性。代码已在https://github.com/ColinLu50/Evade-GPT-Detector 开源。