AI-based code generators are an emerging solution for automatically writing programs starting from descriptions in natural language, by using deep neural networks (Neural Machine Translation, NMT). In particular, code generators have been used for ethical hacking and offensive security testing by generating proof-of-concept attacks. Unfortunately, the evaluation of code generators still faces several issues. The current practice uses output similarity metrics, i.e., automatic metrics that compute the textual similarity of generated code with ground-truth references. However, it is not clear what metric to use, and which metric is most suitable for specific contexts. This work analyzes a large set of output similarity metrics on offensive code generators. We apply the metrics on two state-of-the-art NMT models using two datasets containing offensive assembly and Python code with their descriptions in the English language. We compare the estimates from the automatic metrics with human evaluation and provide practical insights into their strengths and limitations.

翻译：人工智能代码生成器是一种新兴解决方案，通过深度神经网络（神经机器翻译，NMT）从自然语言描述自动编写程序。具体而言，代码生成器已用于道德黑客和防御性安全测试，通过生成概念验证攻击代码。遗憾的是，代码生成器的评估仍面临若干挑战。当前实践采用输出相似性指标，即通过自动计算生成代码与真实参考代码的文本相似度的评估指标。然而，目前尚不明确应使用何种指标，以及哪种指标最适合特定场景。本研究系统分析了攻击性代码生成器的大量输出相似性指标，并基于两个包含攻击性汇编代码和Python代码及其英文描述的数据集，对两种最先进的NMT模型应用这些指标。我们将自动指标的评估结果与人工评估进行比较，并提出关于其优势与局限性的实践见解。