Recent improvements in the quality of the generations by large language models have spurred research into identifying machine-generated text. Such work often presents high-performing detectors. However, humans and machines can produce text in different styles and domains, yet the performance impact of such on machine generated text detection systems remains unclear. In this paper, we audit the classification performance for detecting machine-generated text by evaluating on texts with varying writing styles. We find that classifiers are highly sensitive to stylistic changes and differences in text complexity, and in some cases degrade entirely to random classifiers. We further find that detection systems are particularly susceptible to misclassify easy-to-read texts while they have high performance for complex texts, leading to concerns about the reliability of detection systems. We recommend that future work attends to stylistic factors and reading difficulty levels of human-written and machine-generated text.

翻译：近年来，大型语言模型生成质量的显著提升推动了机器生成文本识别领域的研究。此类研究通常展示出高性能的检测器。然而，人类与机器可能以不同风格和领域生成文本，而这种差异对机器生成文本检测系统性能的影响尚不明确。本文通过评估不同写作风格的文本，对机器生成文本检测的分类性能进行系统性审计。研究发现，分类器对文本风格变化和复杂度差异极为敏感，在某些情况下甚至会完全退化为随机分类器。进一步研究表明，检测系统尤其容易误判易读文本，而对复杂文本则表现出较高性能，这引发了关于检测系统可靠性的担忧。我们建议未来研究应关注人类撰写文本与机器生成文本的风格因素及阅读难度水平。