Large language models have demonstrated great potential to assist programmers in generating code. For such human-AI pair programming scenarios, we empirically demonstrate that while generated code is most often evaluated in terms of their functional correctness (i.e., whether generations pass available unit tests), correctness does not fully capture (e.g., may underestimate) the productivity gains these models may provide. Through a user study with N = 49 experienced programmers, we show that while correctness captures high-value generations, programmers still rate code that fails unit tests as valuable if it reduces the overall effort needed to complete a coding task. Finally, we propose a hybrid metric that combines functional correctness and syntactic similarity and show that it achieves a 14% stronger correlation with value and can therefore better represent real-world gains when evaluating and comparing models.

翻译：大型语言模型已展现出协助程序员生成代码的巨大潜力。针对此类人机协作编程场景，我们通过实证研究发现：虽然生成代码最常以其功能正确性（即能否通过可用单元测试）进行评估，但正确性无法完全反映（如可能低估）这些模型所带来的生产力提升。通过一项涉及49名经验丰富程序员的用户研究，我们证明：虽然正确性能够捕捉高价值生成结果，但程序员仍会将未通过单元测试的代码评为有价值——前提是该代码能减少完成编程任务所需的总工作量。最后，我们提出一种结合功能正确性与句法相似性的混合指标，并证明该指标与价值判断的相关性强了14%，因此能在评估和比较模型时更准确地反映实际收益。