Code completion has become a central task, gaining significant attention with the rise of large language model (LLM)-based tools in software engineering. Although recent advances have greatly improved LLMs' code completion abilities, evaluation methods have not advanced equally. Most current benchmarks focus solely on functional correctness of code completions based on given context, overlooking models' ability to follow user instructions during completion-a common scenario in LLM-assisted programming. To address this limitation, we present the first instruction-guided code completion benchmark, Controllable Code Completion Benchmark (C3-Bench), comprising 2,195 carefully designed completion tasks. Through comprehensive evaluation of over 40 mainstream LLMs across C3-Bench and conventional benchmarks, we reveal substantial gaps in instruction-following capabilities between open-source and advanced proprietary models during code completion tasks. Moreover, we develop a straightforward data synthesis pipeline that leverages Qwen2.5-Coder to generate high-quality instruction-completion pairs for supervised fine-tuning (SFT). The resulting model, Qwen2.5-Coder-C3, achieves state-of-the-art performance on C3-Bench. Our findings provide valuable insights for enhancing LLMs' code completion and instruction-following capabilities, establishing new directions for future research in code LLMs. To facilitate reproducibility and foster further research in code LLMs, we open-source all code, datasets, and models.

翻译：代码补全已成为软件工程领域随着基于大语言模型（LLM）的工具兴起而备受关注的核心任务。尽管近期进展极大提升了LLM的代码补全能力，但评估方法并未同步发展。当前大多数基准测试仅关注基于给定上下文的代码补全功能正确性，忽视了模型在补全过程中遵循用户指令的能力——这是LLM辅助编程中的常见场景。为弥补这一局限，我们提出了首个指令引导的代码补全基准——可控代码补全基准（C3-Bench），包含2,195个精心设计的补全任务。通过对40余个主流LLM在C3-Bench和传统基准上的综合评估，我们揭示了开源模型与先进专有模型在代码补全任务中指令遵循能力的显著差距。此外，我们开发了一个简洁的数据合成流程，利用Qwen2.5-Coder生成高质量的指令-补全对用于监督微调（SFT）。由此得到的模型Qwen2.5-Coder-C3在C3-Bench上实现了最先进的性能。我们的研究结果为增强LLM的代码补全与指令遵循能力提供了宝贵见解，为代码LLM的未来研究确立了新方向。为促进可复现性并推动代码LLM的进一步研究，我们已开源所有代码、数据集和模型。