Stack Overflow is a popular Q&A platform where users ask technical questions and receive answers from a community of experts. Recently, there has been a significant increase in the number of answers generated by ChatGPT, which can lead to incorrect and unreliable information being posted on the site. While Stack Overflow has banned such AI-generated content, detecting whether a post is ChatGPT-generated remains a challenging task. We introduce a novel approach, SOGPTSpotter, that employs Siamese Neural Networks, leveraging the BigBird model and the Triplet loss, to detect ChatGPT-generated answers on Stack Overflow. We use triplets of human answers, reference answers, and ChatGPT answers. Our empirical evaluation reveals that our approach outperforms well-established baselines like GPTZero, DetectGPT, GLTR, BERT, RoBERTa, and GPT-2 in identifying ChatGPT-synthesized Stack Overflow responses. We also conducted an ablation study to show the effectiveness of our model. Additional experiments were conducted to assess various factors, including the impact of text length, the model's robustness against adversarial attacks, and its generalization capabilities across different domains and large language models. We also conducted a real-world case study on Stack Overflow. Using our tool's recommendations, Stack Overflow moderators were able to identify and take down ChatGPT-suspected generated answers, demonstrating the practical applicability and effectiveness of our approach.

翻译：Stack Overflow是一个流行的问答平台，用户在此提出技术问题并从专家社区获得解答。近期，由ChatGPT生成的答案数量显著增加，这可能导致错误和不可靠的信息被发布在该网站上。尽管Stack Overflow已禁止此类AI生成内容，但检测帖子是否由ChatGPT生成仍是一项具有挑战性的任务。我们提出了一种新颖的方法SOGPTSpotter，该方法采用孪生神经网络，利用BigBird模型和三元组损失，来检测Stack Overflow上的ChatGPT生成答案。我们使用包含人工答案、参考答案和ChatGPT答案的三元组进行训练。我们的实证评估表明，在识别ChatGPT合成的Stack Overflow回复方面，我们的方法优于GPTZero、DetectGPT、GLTR、BERT、RoBERTa和GPT-2等成熟基线模型。我们还进行了消融研究以展示模型的有效性。通过额外实验评估了多种因素，包括文本长度的影响、模型对抗对抗攻击的鲁棒性，以及其在不同领域和大语言模型间的泛化能力。此外，我们在Stack Overflow上进行了真实案例研究。借助我们工具的建议，Stack Overflow版主成功识别并移除了疑似ChatGPT生成的答案，这证明了我们方法的实际适用性和有效性。