Prior studies have shown that distinguishing text generated by Large Language Models (LLMs) from human-written one is highly challenging for humans, and often no better than random guessing. To verify the generalizability of this finding across languages and domains, we perform an extensive case study to identify the upper bound of human detection accuracy. Across 16 datasets covering 9 languages and 9 domains, 19 annotators achieved an average detection accuracy of 87.6%, thus challenging previous conclusions. We find that major gaps between human and machine text lie in concreteness, cultural nuances, and diversity. Prompting by explicitly explaining the distinctions in the prompts can partially bridge the gaps in over 50% of the cases. However, we also find that humans do not always prefer human-written text, particularly when they cannot clearly identify its source. We release our dataset, the human labels, and the annotator metadata at https://github.com/xnlp-lab/HumanEval-MGT.

翻译：以往研究表明，人类区分大型语言模型（LLMs）生成的文本与人类撰写的文本极具挑战性，其准确率往往不优于随机猜测。为验证该结论在跨语言与跨领域场景下的普适性，我们通过大规模案例研究探测人类检测准确率的上限。覆盖9种语言与9个领域的16个数据集中，19名标注员实现了平均87.6%的检测准确率，从而对既往结论提出质疑。研究发现，人类文本与机器文本之间的主要差异体现在具象性、文化微妙性及多样性三个维度。在提示语中明确阐释这些差异，能在超过50%的案例中部分弥合差距。然而，我们还发现人类并非总是偏好人类撰写的文本，尤其在无法明确辨识文本来源时。相关数据集、人工标签及标注者元数据已发布于https://github.com/xnlp-lab/HumanEval-MGT。