In recent years, Natural Language Generation (NLG) techniques in AI (e.g., T5, GPT-3, ChatGPT) have shown a massive improvement and are now capable of generating human-like long coherent texts at scale, yielding so-called deepfake texts. This advancement, despite their benefits, can also cause security and privacy issues (e.g., plagiarism, identity obfuscation, disinformation attack). As such, it has become critically important to develop effective, practical, and scalable solutions to differentiate deepfake texts from human-written texts. Toward this challenge, in this work, we investigate how factors such as skill levels and collaborations impact how humans identify deepfake texts, studying three research questions: (1) do collaborative teams detect deepfake texts better than individuals? (2) do expert humans detect deepfake texts better than non-expert humans? (3) what are the factors that maximize the detection performance of humans? We implement these questions on two platforms: (1) non-expert humans or asynchronous teams on Amazon Mechanical Turk (AMT) and (2) expert humans or synchronous teams on the Upwork. By analyzing the detection performance and the factors that affected performance, some of our key findings are: (1) expert humans detect deepfake texts significantly better than non-expert humans, (2) synchronous teams on the Upwork detect deepfake texts significantly better than individuals, while asynchronous teams on the AMT detect deepfake texts weakly better than individuals, and (3) among various error categories, examining coherence and consistency in texts is useful in detecting deepfake texts. In conclusion, our work could inform the design of future tools/framework to improve collaborative human detection of deepfake texts.

翻译：近年来，人工智能中的自然语言生成（NLG）技术（如T5、GPT-3、ChatGPT）取得了巨大进步，能够大规模生成类人化的长连贯文本，从而产生所谓的深度伪造文本。尽管这些技术具有诸多益处，但也可能引发安全与隐私问题（例如，抄袭、身份混淆、虚假信息攻击）。因此，开发有效、实用且可扩展的解决方案来区分深度伪造文本与人类撰写的文本变得至关重要。针对这一挑战，本研究探讨了技能水平与协作等因素如何影响人类识别深度伪造文本的能力，并提出三个研究问题：（1）协作团队在检测深度伪造文本方面是否优于个体？（2）专家人类在检测深度伪造文本方面是否优于非专家人类？（3）最大化人类检测性能的因素有哪些？我们在两个平台上实施这些问题：（1）在Amazon Mechanical Turk（AMT）上招募非专家人类或异步团队；（2）在Upwork上招募专家人类或同步团队。通过分析检测性能及其影响因素，我们得出以下关键发现：（1）专家人类检测深度伪造文本的能力显著优于非专家人类；（2）Upwork上的同步团队检测深度伪造文本的能力显著优于个体，而AMT上的异步团队检测能力略优于个体；（3）在各种错误类别中，检查文本的连贯性与一致性有助于检测深度伪造文本。总之，本研究可为未来设计改进人类协作检测深度伪造文本的工具或框架提供参考。