Advances in Large Language Models (e.g., GPT-4, LLaMA) have improved the generation of coherent sentences resembling human writing on a large scale, resulting in the creation of so-called deepfake texts. However, this progress poses security and privacy concerns, necessitating effective solutions for distinguishing deepfake texts from human-written ones. Although prior works studied humans' ability to detect deepfake texts, none has examined whether "collaboration" among humans improves the detection of deepfake texts. In this study, to address this gap of understanding on deepfake texts, we conducted experiments with two groups: (1) nonexpert individuals from the AMT platform and (2) writing experts from the Upwork platform. The results demonstrate that collaboration among humans can potentially improve the detection of deepfake texts for both groups, increasing detection accuracies by 6.36% for non-experts and 12.76% for experts, respectively, compared to individuals' detection accuracies. We further analyze the explanations that humans used for detecting a piece of text as deepfake text, and find that the strongest indicator of deepfake texts is their lack of coherence and consistency. Our study provides useful insights for future tools and framework designs to facilitate the collaborative human detection of deepfake texts. The experiment datasets and AMT implementations are available at: https://github.com/huashen218/llm-deepfake-human-study.git

翻译：大型语言模型（如GPT-4、LLaMA）的进步使得大规模生成类似人类写作的连贯句子成为可能，从而产生了所谓的深度伪造文本。然而，这一进展引发了安全与隐私方面的担忧，亟需有效的解决方案来区分深度伪造文本与人类撰写的文本。尽管已有研究探讨了人类单独检测深度伪造文本的能力，但尚无研究考察人类之间的“协作”是否能提升深度伪造文本的检测效果。为弥补这一认知空白，本研究分别针对两组对象开展了实验：（1）来自AMT平台的非专业个体；（2）来自Upwork平台的写作专家。结果表明，人类协作能显著提升两组人员的检测准确性——非专业人员组提高6.36%，专家组成提高12.76%，均优于个体单独检测的表现。我们进一步分析了人类用于判定某段文本为深度伪造文本的依据，发现深度伪造文本最显著的标志是其缺乏连贯性与一致性。本研究为未来设计促进人类协作检测深度伪造文本的工具与框架提供了有益启示。实验数据集及AMT实施代码均可在以下地址获取：https://github.com/huashen218/llm-deepfake-human-study.git