Large Language Models (LLMs) such as GPT-4, Claude and LLaMA have shown impressive performance in code generation, typically evaluated using benchmarks (e.g., HumanEval). However, effective code generation requires models to understand and apply a wide range of language concepts. If the concepts exercised in benchmarks are not representative of those used in real-world projects, evaluations may yield incomplete. Despite this concern, the representativeness of code concepts in benchmarks has not been systematically examined. To address this gap, we present the first empirical study that analyzes the representativeness of code generation benchmarks through the lens of Knowledge Units (KUs) - cohesive sets of programming language capabilities provided by language constructs and APIs. We analyze KU coverage in two widely used Python benchmarks, HumanEval and MBPP, and compare them with 30 real-world Python projects. Our results show that each benchmark covers only half of the identified 20 KUs, whereas projects exercise all KUs with relatively balanced distributions. In contrast, benchmark tasks exhibit highly skewed KU distributions. To mitigate this misalignment, we propose a prompt-based LLM framework that synthesizes KU-based tasks to rebalance benchmark KU distributions and better align them with real-world usage. Using this framework, we generate 440 new tasks and augment existing benchmarks. The augmented benchmarks substantially improve KU coverage and achieve over a 60% improvement in distributional alignment. Evaluations of state-of-the-art LLMs on these augmented benchmarks reveal consistent and statistically significant performance drops (12.54-44.82%), indicating that existing benchmarks overestimate LLM performance due to their limited KU coverage. Our findings provide actionable guidance for building more realistic evaluations of LLM code-generation capabilities.

翻译：以GPT-4、Claude和LLaMA为代表的大型语言模型在代码生成任务中展现出卓越性能，其评估通常依赖HumanEval等基准测试集。然而，有效的代码生成要求模型理解并运用广泛的语言概念。若基准测试中考察的概念未能充分反映实际项目中的概念分布，评估结果可能具有局限性。尽管这一问题至关重要，代码生成基准中编程概念的代表性尚未得到系统性检验。为填补这一空白，我们首次通过知识单元的视角对代码生成基准的代表性展开实证研究——知识单元是由语言构造和API提供的、具有内在一致性的编程能力集合。我们分析了两个广泛使用的Python基准测试集（HumanEval与MBPP）的知识单元覆盖情况，并将其与30个真实Python项目进行对比。结果显示：每个基准测试集仅覆盖已识别的20个知识单元中的半数，而实际项目则覆盖全部知识单元且分布相对均衡；相比之下，基准测试任务呈现高度偏态的知识单元分布。为缓解这种偏差，我们提出一种基于提示词的大型语言模型框架，通过合成基于知识单元的任务来重新平衡基准测试的知识单元分布，使其更贴近实际使用场景。利用该框架，我们生成了440项新任务并对现有基准进行增强。增强后的基准在知识单元覆盖率上显著提升，分布对齐度改善超过60%。在最先进的大型语言模型上对增强基准进行评估后，模型性能出现一致且统计显著的下降（12.54-44.82%），这表明现有基准因知识单元覆盖不足而高估了大型语言模型的真实性能。本研究为构建更贴近现实的大型语言模型代码生成能力评估体系提供了可操作的指导。