As Large Language Models (LLMs) have become capable of generating long and descriptive code summaries, accurate and reliable evaluation of factual consistency has become a critical challenge. However, previous evaluation methods are primarily designed for short summaries of isolated code snippets. Consequently, they struggle to provide fine-grained evaluation of multi-sentence functionalities and fail to accurately assess dependency context commonly found in real-world code summaries. To address this, we propose ReFEree, a reference-free and fine-grained method for evaluating factual consistency in real-world code summaries. We define factual inconsistency criteria specific to code summaries and evaluate them at the segment level using these criteria along with dependency information. These segment-level results are then aggregated into a fine-grained score. We construct a code summarization benchmark with human-annotated factual consistency labels. The evaluation results demonstrate that ReFEree achieves the highest correlation with human judgment among 13 baselines, improving 15-18% over the previous state-of-the-art. Our code and data are available at https://github.com/bsy99615/ReFEree.git.

翻译：随着大语言模型（LLMs）能够生成长篇幅、描述性的代码总结，事实一致性的准确可靠评估已成为关键挑战。然而，现有评估方法主要针对孤立代码片段的简短总结而设计，因此难以对多句功能进行细粒度评估，也无法准确评估真实世界代码总结中常见的依赖上下文。为解决此问题，我们提出ReFEree——一种面向真实世界代码总结中事实一致性的无参考细粒度评估方法。我们定义了专用于代码总结的事实不一致性标准，并利用这些标准结合依赖信息在片段级别进行评估，再将片段级结果聚合为细粒度得分。我们构建了一个带有人工标注事实一致性标签的代码总结基准数据集。评估结果表明，在13个基线方法中，ReFEree与人工判断的相关性最高，较先前最优方法提升了15-18%。我们的代码和数据已在https://github.com/bsy99615/ReFEree.git公开。