Since its launch in November 2022, ChatGPT has gained popularity among users, especially programmers who use it as a tool to solve development problems. However, while offering a practical solution to programming problems, ChatGPT should be mainly used as a supporting tool (e.g., in software education) rather than as a replacement for the human being. Thus, detecting automatically generated source code by ChatGPT is necessary, and tools for identifying AI-generated content may need to be adapted to work effectively with source code. This paper presents an empirical study to investigate the feasibility of automated identification of AI-generated code snippets, and the factors that influence this ability. To this end, we propose a novel approach called GPTSniffer, which builds on top of CodeBERT to detect source code written by AI. The results show that GPTSniffer can accurately classify whether code is human-written or AI-generated, and outperforms two baselines, GPTZero and OpenAI Text Classifier. Also, the study shows how similar training data or a classification context with paired snippets helps to boost classification performances.

翻译：摘要：自2022年11月发布以来，ChatGPT在用户中广受欢迎，尤其是将其作为解决开发问题工具的开发者。然而，尽管ChatGPT为编程问题提供了实用的解决方案，它主要应作为辅助工具（例如在软件教育中）而非人类替代品使用。因此，自动检测由ChatGPT生成的源代码具有必要性，而用于识别AI生成内容的工具可能需要针对源代码进行适配。本文开展实证研究，以探究自动识别AI生成代码片段的可行性及其影响因素。为此，我们提出了一种名为GPTSniffer的新方法，该方法基于CodeBERT构建，用于检测AI编写的源代码。结果表明，GPTSniffer能准确区分代码是由人类编写还是AI生成，其性能优于GPTZero和OpenAI文本分类器这两个基线模型。此外，研究还表明，采用相似训练数据或结合配对片段的分类上下文有助于提升分类性能。