Instruction-tuned Language Models ILMs have become essential components of modern AI systems, demonstrating exceptional versatility across a wide range of natural language and reasoning tasks. Among their most impactful applications is code generation, where ILMs--commonly referred to as Code Language Models CLMs--have demonstrated remarkable capability. This strength stems from their defining feature: the use of explicit task instructions during fine-tuning, which enables them to bridge natural language and code by translating human intent into executable code. While much of their progress has been driven by advances in scaling laws and training methodologies, one critical aspect remains underexplored--the impact of system prompts on the performance of both general-purpose ILMs and specialized CLMs when instantiated to assist users with code generation activities. In this study, we take a first step toward bridging this gap by systematically evaluating how system prompts of varying instructional detail, along with model scale, prompting strategy, and programming language, affect ILMs and CLMs in code generation tasks. Our evaluation framework, spanning 120 model configurations, reveals that (1) the influence of system prompts increases with model scale; (2) few-shot prompting reduces this effect compared to zero-shot; and (3) programming language matters, with Java showing greater sensitivity to system prompt variations than Python.

翻译：指令调优语言模型已成为现代人工智能系统的核心组件，在广泛的自然语言与推理任务中展现出卓越的通用性。其中最具影响力的应用之一是代码生成领域，指令调优语言模型（通常称为代码语言模型）在此表现出非凡能力。这种优势源于其定义性特征：在微调过程中使用显式任务指令，使其能够通过将人类意图转化为可执行代码来桥接自然语言与编程语言。尽管其进展主要受规模定律与训练方法学进步的驱动，但一个关键维度仍未得到充分探索——当实例化用于辅助用户进行代码生成活动时，系统提示对通用指令调优语言模型与专用代码语言模型性能的影响。本研究通过系统评估不同详细程度的系统提示（结合模型规模、提示策略及编程语言）如何影响指令调优语言模型在代码生成任务中的表现，首次尝试填补这一研究空白。我们的评估框架涵盖120种模型配置，结果表明：（1）系统提示的影响随模型规模增大而增强；（2）与零样本提示相比，少样本提示会减弱这种影响；（3）编程语言具有显著影响，Java对系统提示变化的敏感性高于Python。