Large Language Models (LLMs) possess impressive capabilities to generate meaningful code snippets given natural language intents in zero-shot, i.e., without the need for specific fine-tuning. In the perspective of unleashing their full potential, prior work has demonstrated the benefits of fine-tuning the models to task-specific data. However, fine-tuning process demands heavy computational costs and is intractable when resources are scarce, especially for models with billions of parameters. In light of these challenges, previous studies explored In-Context Learning (ICL) as an effective strategy to generate contextually appropriate code without fine-tuning. However, it operates at inference time and does not involve learning task-specific parameters, potentially limiting the model's performance on downstream tasks. In this context, we foresee that Parameter-Efficient Fine-Tuning (PEFT) techniques carry a high potential for efficiently specializing LLMs to task-specific data. In this paper, we deliver a comprehensive study of LLMs with the impact of PEFT techniques under the automated code generation scenario. Our experimental results reveal the superiority and potential of such techniques over ICL on a wide range of LLMs in reducing the computational burden and improving performance. Therefore, the study opens opportunities for broader applications of PEFT in software engineering scenarios.

翻译：大语言模型（LLMs）具有令人印象深刻的能力，能在零样本条件下根据自然语言意图生成有意义的代码片段，而无需进行特定的微调。为了充分发挥其潜力，先前的研究已证明了将模型微调到特定任务数据的益处。然而，微调过程需要高昂的计算成本，并且在资源稀缺时难以处理，尤其是对于拥有数十亿参数的模型而言。鉴于这些挑战，先前的研究探索了上下文学习作为一种无需微调即可生成符合上下文代码的有效策略。然而，它仅在推理时运行，不涉及学习特定任务参数，这可能会限制模型在下游任务中的性能。在此背景下，我们预见到参数高效微调技术具有高效地将大语言模型专门化到特定任务数据的巨大潜力。在本文中，我们基于自动代码生成场景，对大语言模型受参数高效微调技术影响的情况进行了全面研究。我们的实验结果揭示了这些技术相较于上下文学习在多种大语言模型上的优越性和潜力，能减少计算负担并提升性能。因此，这项研究为参数高效微调技术在软件工程场景中的更广泛应用开辟了机会。