Thanks to the rapid adoption of AI code assistants powered by large language models (LLMs), industry codebases are, increasingly, a hybrid of AI- and human-authored code. For risk management and productivity analysis purposes, it is crucial to enable fine-grained location detection of AI-generated code. To develop algorithms for this task, quality benchmarks are needed to assess performance. However, existing benchmarks tend to comprise academic, LeetCode-style problems and presume a code snippet is either completely human-authored or completely AI-authored, which is not reflective of the diverse intents and styles of industry codebases utilizing AI code assistants. To fill these gaps, we introduce HybridCodeAuthorship, a novel benchmark of Python code files with interleaved human- and AI-authored lines of code to simulate authentic utilization of AI code assistants. In this paper, we first present our dataset construction pipeline, which leverages CodeSearchNet, a massive collection of links to open sourced repositories on GitHub. We then benchmark the performance of two state-of-the-art AI-generated code detection algorithms at both the line- and chunk-level. Experimental results demonstrate that HybridCodeAuthorship is a challenging benchmark with a top-scoring algorithm, AIGCode Detector, obtaining a highest F1 score of 0.48 and 0.56 on chunk-level and line-level code detection tasks, respectively.

翻译：得益于基于大型语言模型（LLMs）的AI代码助手的快速普及，工业级代码库正日益成为AI生成代码与人类编写代码的混合体。出于风险管理和生产力分析的目的，实现AI生成代码的细粒度位置检测至关重要。为开发此类检测算法，需要高质量的基准来评估性能。然而，现有基准数据集通常包含学术性、LeetCode风格的题目，并且假设代码片段要么完全由人类编写，要么完全由AI生成，这未能反映工业代码库在使用AI代码助手时的多样化意图与风格。为填补这些空白，我们提出了HybridCodeAuthorship，一个新颖的Python代码文件基准数据集，其特点是在行级别交替混合人类编写与AI生成的代码，以模拟AI代码助手的真实使用场景。本文首先介绍了我们的数据集构建流程，该流程利用了CodeSearchNet（一个包含GitHub上开源仓库链接的大规模集合）。随后，我们在行级和片段级两种粒度上，对两种最先进的AI生成代码检测算法进行了基准测试。实验结果表明，HybridCodeAuthorship是一个具有挑战性的基准，其中得分最高的算法AIGCode Detector在片段级和行级代码检测任务上分别取得了0.48和0.56的最高F1分数。