With the increasing popularity of LLM-based code completers, like GitHub Copilot, the interest in automatically detecting AI-generated code is also increasing-in particular in contexts where the use of LLMs to program is forbidden by policy due to security, intellectual property, or ethical concerns.We introduce a novel technique for AI code stylometry, i.e., the ability to distinguish code generated by LLMs from code written by humans, based on a transformer-based encoder classifier. Differently from previous work, our classifier is capable of detecting AI-written code across 10 different programming languages with a single machine learning model, maintaining high average accuracy across all languages (84.1% $\pm$ 3.8%).Together with the classifier we also release H-AIRosettaMP, a novel open dataset for AI code stylometry tasks, consisting of 121 247 code snippets in 10 popular programming languages, labeled as either human-written or AI-generated. The experimental pipeline (dataset, training code, resulting models) is the first fully reproducible one for the AI code stylometry task. Most notably our experiments rely only on open LLMs, rather than on proprietary/closed ones like ChatGPT.

翻译：随着基于LLM的代码补全工具（如GitHub Copilot）日益普及，对AI生成代码的自动检测需求也持续增长——特别是在因安全、知识产权或伦理考量而禁止使用LLM编程的政策环境中。我们提出一种创新的AI代码风格测量技术，即基于Transformer编码器分类器来区分LLM生成的代码与人类编写的代码。与先前研究不同，我们的分类器能够通过单一机器学习模型检测10种不同编程语言中的AI生成代码，并在所有语言中保持较高的平均准确率（84.1% ± 3.8%）。除分类器外，我们还发布了H-AIRosettaMP——一个用于AI代码风格测量任务的新型开源数据集，包含10种流行编程语言的121,247个代码片段，并标注为人类编写或AI生成。该实验流程（数据集、训练代码、生成模型）是首个完全可复现的AI代码风格测量解决方案。值得注意的是，我们的实验仅依赖开源LLM，而非ChatGPT等专有/闭源模型。