Code Large Language Models (Code LLMs) are being increasingly employed in real-life applications, so evaluating them is critical. While the conventional accuracy evaluates the performance of Code LLMs on a set of individual tasks, their self-consistency across different tasks is overlooked. Intuitively, a trustworthy model should be self-consistent when generating natural language specifications for its own code and generating code for its own specifications. Failure to preserve self-consistency reveals a lack of understanding of the shared semantics underlying natural language and programming language, and therefore undermines the trustworthiness of a model. In this paper, we first formally define the self-consistency of Code LLMs and then design a framework, IdentityChain, which effectively and efficiently evaluates the self-consistency and conventional accuracy of a model at the same time. We study eleven Code LLMs and show that they fail to preserve self-consistency, which is indeed a distinct aspect from conventional accuracy. Furthermore, we show that IdentityChain can be used as a model debugging tool to expose weaknesses of Code LLMs by demonstrating three major weaknesses that we identify in current models using IdentityChain. Our code is available at https://github.com/marcusm117/IdentityChain.

翻译：代码大语言模型（Code LLMs）正越来越多地应用于实际场景，因此对其进行评估至关重要。传统准确率衡量了代码大语言模型在一组独立任务上的表现，但其在不同任务间的自我一致性却被忽略。直观而言，一个可信赖的模型在为自身代码生成自然语言规范以及为自身规范生成代码时，应保持自我一致性。若无法维持自我一致性，则表明模型对自然语言与编程语言共享的语义理解不足，从而削弱其可信赖性。本文首先形式化定义了代码大语言模型的自我一致性，随后设计了一个名为IdentityChain的框架，该框架能够高效且有效地同时评估模型的自我一致性与传统准确率。我们研究了十一个代码大语言模型，结果表明它们未能保持自我一致性，而这确实是与传统准确率不同的独立维度。此外，我们展示了IdentityChain可作为模型调试工具，通过揭示当前模型中存在的三大主要弱点（基于IdentityChain识别结果）来暴露其不足。我们的代码已开源在https://github.com/marcusm117/IdentityChain。