Code Large Language Models (Code LLMs) are being increasingly employed in real-life applications, so evaluating them is critical. While the conventional accuracy evaluates the performance of Code LLMs on a set of individual tasks, their self-consistency across different tasks is overlooked. Intuitively, a trustworthy model should be self-consistent when generating natural language specifications for its own code and generating code for its own specifications. Failure to preserve self-consistency reveals a lack of understanding of the shared semantics underlying natural language and programming language, and therefore undermines the trustworthiness of a model. In this paper, we first formally define the self-consistency of Code LLMs and then design a framework, IdentityChain, which effectively and efficiently evaluates the self-consistency and conventional accuracy of a model at the same time. We study eleven Code LLMs and show that they fail to preserve self-consistency, which is indeed a distinct aspect from conventional accuracy. Furthermore, we show that IdentityChain can be used as a model debugging tool to expose weaknesses of Code LLMs by demonstrating three major weaknesses that we identify in current models using IdentityChain. Our code is available at https://github.com/marcusm117/IdentityChain.

翻译：代码大语言模型正日益广泛应用于实际场景，因此对其评估至关重要。传统准确率仅衡量模型在单一任务上的表现，却忽视了跨任务间的自我一致性。直观而言，可信赖的模型在为其代码生成自然语言规范以及根据规范生成代码时，应保持自我一致性。若无法维持自我一致性，则表明模型缺乏对自然语言与编程语言共享语义的理解，从而削弱其可信度。本文首先正式定义代码大语言模型的自我一致性，随后设计IdentityChain框架，该框架能同时高效评估模型的自我一致性与传统准确率。我们研究了11个代码大语言模型，发现它们均未能保持自我一致性，而这恰恰是与传统准确率截然不同的评估维度。此外，我们证明IdentityChain可作为模型调试工具，通过揭示当前模型存在的三大主要缺陷来暴露其弱点。相关代码已开源至https://github.com/marcusm117/IdentityChain。