Evaluating the correctness of code generated by AI is a challenging open problem. In this paper, we propose a fully automated method, named ACCA, to evaluate the correctness of AI-generated code for security purposes. The method uses symbolic execution to assess whether the AI-generated code behaves as a reference implementation. We use ACCA to assess four state-of-the-art models trained to generate security-oriented assembly code and compare the results of the evaluation with different baseline solutions, including output similarity metrics, widely used in the field, and the well-known ChatGPT, the AI-powered language model developed by OpenAI. Our experiments show that our method outperforms the baseline solutions and assesses the correctness of the AI-generated code similar to the human-based evaluation, which is considered the ground truth for the assessment in the field. Moreover, ACCA has a very strong correlation with the human evaluation (Pearson's correlation coefficient r=0.84 on average). Finally, since it is a fully automated solution that does not require any human intervention, the proposed method performs the assessment of every code snippet in ~0.17s on average, which is definitely lower than the average time required by human analysts to manually inspect the code, based on our experience.

翻译：评估AI生成代码的正确性是一项具有挑战性的开放性问题。本文提出了一种名为ACCA的全自动方法，用于评估面向安全用途的AI生成代码的正确性。该方法采用符号执行技术来判定AI生成代码是否与参考实现的行为一致。我们利用ACCA评估了四个用于生成面向安全汇编代码的最先进模型，并将评估结果与多种基线方案进行了对比，包括该领域广泛使用的输出相似度指标以及著名的OpenAI开发的AI语言模型ChatGPT。实验表明，我们的方法优于基线方案，其评估AI生成代码正确性的表现与人工评估（该领域公认的评估标准）相当。此外，ACCA与人工评估结果具有极强的相关性（平均皮尔逊相关系数r=0.84）。最后，作为无需任何人工干预的全自动解决方案，该方法平均每段代码的评估时间约为0.17秒，根据我们的经验，这远低于人工分析师手动审查代码所需的平均时间。