Large Language Models (LLMs) have become integral to various software engineering tasks, including code generation, bug detection, and repair. To evaluate model performance in these domains, numerous bug benchmarks containing real-world bugs from software projects have been developed. However, a growing concern within the software engineering community is that these benchmarks may not reliably reflect true LLM performance due to the risk of data leakage. Despite this concern, limited research has been conducted to quantify the impact of potential leakage. In this paper, we systematically evaluate popular LLMs to assess their susceptibility to data leakage from widely used bug benchmarks. To identify potential leakage, we use multiple metrics, including a study of benchmark membership within commonly used training datasets, as well as analyses of negative log-likelihood and n-gram accuracy. Our findings show that certain models, in particular codegen-multi, exhibit significant evidence of memorization in widely used benchmarks like Defects4J, while newer models trained on larger datasets like LLaMa 3.1 exhibit limited signs of leakage. These results highlight the need for careful benchmark selection and the adoption of robust metrics to adequately assess models capabilities.

翻译：大型语言模型（LLMs）已成为各类软件工程任务不可或缺的组成部分，涵盖代码生成、缺陷检测与修复等领域。为评估模型在这些领域的性能，研究者已开发出众多包含真实软件项目缺陷的漏洞基准测试集。然而，软件工程界日益担忧的是，由于存在数据泄露风险，这些基准测试可能无法可靠反映大型语言模型的真实性能。尽管存在这种担忧，目前量化潜在泄露影响的研究仍十分有限。本文通过系统评估主流大型语言模型，探究其受广泛使用的缺陷基准测试数据泄露影响的程度。为识别潜在泄露，我们采用多种度量指标，包括对常用训练数据集中基准测试成员资格的研究，以及负对数似然与n-gram准确率的分析。研究发现，特定模型（尤其是codegen-multi）在Defects4J等广泛使用的基准测试中表现出显著记忆迹象，而基于更大数据集训练的新模型（如LLaMa 3.1）则显示出有限的泄露特征。这些结果凸显了谨慎选择基准测试并采用鲁棒度量指标以充分评估模型能力的必要性。