This work proposes a training-free approach for the detection of LLMs-generated codes, mitigating the risks associated with their indiscriminate usage. To the best of our knowledge, our research is the first to investigate zero-shot detection techniques applied to code generated by advanced black-box LLMs like ChatGPT. Firstly, we find that existing training-based or zero-shot text detectors are ineffective in detecting code, likely due to the unique statistical properties found in code structures. We then modify the previous zero-shot text detection method, DetectGPT (Mitchell et al., 2023) by utilizing a surrogate white-box model to estimate the probability of the rightmost tokens, allowing us to identify code snippets generated by language models. Through extensive experiments conducted on the python codes of the CodeContest and APPS dataset, our approach demonstrates its effectiveness by achieving state-of-the-art detection results on text-davinci-003, GPT-3.5, and GPT-4 models. Moreover, our method exhibits robustness against revision attacks and generalizes well to Java codes. We also find that the smaller code language model like PolyCoder-160M performs as a universal code detector, outperforming the billion-scale counterpart. The codes will be available at https://github.com/ Xianjun-Yang/Code_detection.git

翻译：本工作提出了一种无需训练的检测方法，用于识别由大语言模型生成的代码，以减轻其滥用带来的风险。据我们所知，本研究首次探索了针对ChatGPT等高级黑盒大语言模型生成代码的零样本检测技术。首先，我们发现现有的基于训练或零样本文本检测器在检测代码方面效果不佳，这可能是由于代码结构中独特的统计特性所致。随后，我们改进了先前零样本文本检测方法DetectGPT（Mitchell等人，2023），利用替代白盒模型估计最右侧词元的概率，从而识别语言模型生成的代码片段。通过在CodeContest和APPS数据集中的Python代码上进行大量实验，我们的方法在text-davinci-003、GPT-3.5和GPT-4模型上取得了最先进的检测结果，证明了其有效性。此外，该方法对修订攻击具有鲁棒性，并能良好地泛化至Java代码。我们还发现，像PolyCoder-160M这样较小的代码语言模型可作为通用代码检测器，其性能优于十亿规模的同类模型。相关代码将发布于https://github.com/Xianjun-Yang/Code_detection.git