Recent advancements in large language models (LLMs) have automated various software engineering tasks, with benchmarks emerging to evaluate their capabilities. However, for adaptation, a critical activity during code reuse, there is no benchmark to assess LLMs' performance, leaving their practical utility in this area unclear. To fill this gap, we propose AdaptEval, a benchmark designed to evaluate LLMs on code snippet adaptation. Unlike existing benchmarks, AdaptEval incorporates the following three distinctive features: First, Practical Context. Tasks in AdaptEval are derived from developers' practices, preserving rich contextual information from Stack Overflow and GitHub communities. Second, Multi-granularity Annotation. Each task is annotated with requirements at both task and adaptation levels, supporting the evaluation of LLMs across diverse adaptation scenarios. Third, Fine-grained Evaluation. AdaptEval includes a two-tier testing framework combining adaptation-level and function-level tests, which enables evaluating LLMs' performance across various individual adaptations. Based on AdaptEval, we conduct the first empirical study to evaluate six instruction-tuned LLMs and especially three reasoning LLMs on code snippet adaptation. Experimental results demonstrate that AdaptEval enables the assessment of LLMs' adaptation capabilities from various perspectives. It also provides critical insights into their current limitations, particularly their struggle to follow explicit instructions. We hope AdaptEval can facilitate further investigation and enhancement of LLMs' capabilities in code snippet adaptation, supporting their real-world applications.

翻译：近年来，大语言模型（LLMs）的进展已自动化了多种软件工程任务，相应的基准也随之出现以评估其能力。然而，对于代码复用过程中的关键活动——适应，目前尚无基准来评估LLMs的性能，导致其在该领域的实际效用尚不明确。为填补这一空白，我们提出了AdaptEval，一个专为评估LLMs在代码片段适应任务上而设计的基准。与现有基准不同，AdaptEval包含以下三个显著特点：第一，实际上下文。AdaptEval中的任务源自开发者的实践，保留了来自Stack Overflow和GitHub社区的丰富上下文信息。第二，多粒度标注。每个任务均在任务级别和适应级别上进行了需求标注，支持评估LLMs在多样化适应场景下的表现。第三，细粒度评估。AdaptEval包含一个结合适应级别和函数级别的双层测试框架，能够评估LLMs在各种独立适应任务上的性能。基于AdaptEval，我们进行了首次实证研究，评估了六种指令调优的LLMs，特别是三种推理LLMs在代码片段适应任务上的表现。实验结果表明，AdaptEval能够从多个角度评估LLMs的适应能力，并揭示了其当前的关键局限性，尤其是在遵循明确指令方面的困难。我们希望AdaptEval能够促进对LLMs在代码片段适应能力上的进一步研究和提升，支持其在实际应用中的发展。