AI-powered programming language generation (PLG) models have gained increasing attention due to their ability to generate source code of programs in a few seconds with a plain program description. Despite their remarkable performance, many concerns are raised over the potential risks of their development and deployment, such as legal issues of copyright infringement induced by training usage of licensed code, and malicious consequences due to the unregulated use of these models. In this paper, we present the first-of-its-kind study to systematically investigate the accountability of PLG models from the perspectives of both model development and deployment. In particular, we develop a holistic framework not only to audit the training data usage of PLG models, but also to identify neural code generated by PLG models as well as determine its attribution to a source model. To this end, we propose using membership inference to audit whether a code snippet used is in the PLG model's training data. In addition, we propose a learning-based method to distinguish between human-written code and neural code. In neural code attribution, through both empirical and theoretical analysis, we show that it is impossible to reliably attribute the generation of one code snippet to one model. We then propose two feasible alternative methods: one is to attribute one neural code snippet to one of the candidate PLG models, and the other is to verify whether a set of neural code snippets can be attributed to a given PLG model. The proposed framework thoroughly examines the accountability of PLG models which are verified by extensive experiments. The implementations of our proposed framework are also encapsulated into a new artifact, named CodeForensic, to foster further research.

翻译：人工智能驱动的编程语言生成（PLG）模型因其能通过简单程序描述在数秒内生成源代码的能力而日益受到关注。尽管其性能卓越，但其开发与部署过程中的潜在风险引发了诸多担忧，例如因使用许可代码进行训练而导致的版权侵权法律问题，以及这些模型不受监管使用所引发的恶意后果。本文首次系统性地从模型开发和部署两个角度研究了PLG模型的可问责性。具体而言，我们构建了一个全面的框架，不仅用于审计PLG模型的训练数据使用情况，还用于识别PLG模型生成的神经代码，并确定其归因于源模型。为此，我们提出利用成员推理来审计某段代码是否被用于PLG模型的训练数据。此外，我们提出了一种基于学习的方法来区分人类编写的代码和神经代码。在神经代码归因方面，通过实证与理论分析，我们表明将某段代码的生成可靠地归因于某一模型是不可能的。随后，我们提出了两种可行的替代方法：一种是将一段神经代码归因于候选PLG模型之一，另一种是验证一组神经代码片段是否可被归因于某个给定的PLG模型。所提出的框架全面检验了PLG模型的可问责性，并通过大量实验进行了验证。我们还将该框架的实现封装为一个名为CodeForensic的新工具，以促进进一步的研究。