Nowadays large language models (LLMs) have shown revolutionary power in a variety of natural language processing (NLP) tasks such as text classification, sentiment analysis, language translation, and question-answering. In this way, detecting machine-generated texts (MGTs) is becoming increasingly important as LLMs become more advanced and prevalent. These models can generate human-like language that can be difficult to distinguish from text written by a human, which raises concerns about authenticity, accountability, and potential bias. However, existing detection methods against MGTs are evaluated under different model architectures, datasets, and experimental settings, resulting in a lack of a comprehensive evaluation framework across different methodologies In this paper, we fill this gap by proposing the first benchmark framework for MGT detection, named MGTBench. Extensive evaluations on public datasets with curated answers generated by ChatGPT (the most representative and powerful LLMs thus far) show that most of the current detection methods perform less satisfactorily against MGTs. An exceptional case is ChatGPT Detector, which is trained with ChatGPT-generated texts and shows great performance in detecting MGTs. Nonetheless, we note that only a small fraction of adversarial-crafted perturbations on MGTs can evade the ChatGPT Detector, thus highlighting the need for more robust MGT detection methods. We envision that MGTBench will serve as a benchmark tool to accelerate future investigations involving the evaluation of state-of-the-art MGT detection methods on their respective datasets and the development of more advanced MGT detection methods. Our source code and datasets are available at https://github.com/xinleihe/MGTBench.

翻译：当前，大语言模型在文本分类、情感分析、语言翻译及问答等多种自然语言处理任务中展现出革命性能力。随着大语言模型日趋先进与普及，检测机器生成文本的重要性与日俱增。这类模型能生成近似人类语言风格的文本，使其与人类撰写的文本难以区分，进而引发真实性、责任归属及潜在偏见等担忧。然而，现有机器生成文本检测方法在不同模型架构、数据集与实验设置下进行评估，缺乏跨方法的综合性评估框架。为填补这一空白，本文提出首个机器生成文本检测基准框架MGTBench。基于当前最具代表性与能力最强的ChatGPT生成的整理答案，在公开数据集上开展的大规模评估表明，多数现有检测方法对机器生成文本的表现不尽如人意。其中，经ChatGPT生成文本训练的ChatGPT Detector表现优异，成为例外。但值得注意的是，仅需对机器生成文本施加少量对抗性扰动即可规避该检测器，凸显了开发更鲁棒机器生成文本检测方法的迫切需求。我们期望MGTBench能成为加速未来研究的基准工具，助力评估最新机器生成文本检测方法在其对应数据集上的表现，并推动更先进检测方法的发展。源代码与数据集已开源至 https://github.com/xinleihe/MGTBench。