With the advent of fluent generative language models that can produce convincing utterances very similar to those written by humans, distinguishing whether a piece of text is machine-generated or human-written becomes more challenging and more important, as such models could be used to spread misinformation, fake news, fake reviews and to mimic certain authors and figures. To this end, there have been a slew of methods proposed to detect machine-generated text. Most of these methods need access to the logits of the target model or need the ability to sample from the target. One such black-box detection method relies on the observation that generated text is locally optimal under the likelihood function of the generator, while human-written text is not. We find that overall, smaller and partially-trained models are better universal text detectors: they can more precisely detect text generated from both small and larger models. Interestingly, we find that whether the detector and generator were trained on the same data is not critically important to the detection success. For instance the OPT-125M model has an AUC of 0.81 in detecting ChatGPT generations, whereas a larger model from the GPT family, GPTJ-6B, has AUC of 0.45.

翻译：随着能够生成与人类写作极为相似的流畅语句的生成式语言模型的出现，区分一段文本是由机器生成还是人类撰写变得更具挑战性且更为重要，因为此类模型可能被用于传播错误信息、虚假新闻、虚假评论，以及模仿特定作者和人物。为此，研究者提出了众多检测机器生成文本的方法。大多数方法需要访问目标模型的逻辑值（logits）或具备从目标模型采样的能力。一种黑盒检测方法基于以下观察：生成文本在生成器的似然函数下具有局部最优性，而人类撰写的文本则不具备这一特征。我们发现，总体而言，更小且部分训练完成的模型是更优秀的通用文本检测器：它们能够更精确地检测由小型和大型模型生成的文本。有趣的是，我们发现检测器与生成器是否在相同数据上训练对检测成功与否并不关键。例如，OPT-125M模型在检测ChatGPT生成的文本时AUC达到0.81，而GPT系列中的较大模型GPTJ-6B的AUC仅为0.45。