In this work, we evaluate 10 open-source instructed LLMs on four representative code comprehension and generation tasks. We have the following main findings. First, for the zero-shot setting, instructed LLMs are very competitive on code comprehension and generation tasks and sometimes even better than small SOTA models specifically fine-tuned on each downstream task. We also find that larger instructed LLMs are not always better on code-related tasks. Second, for the few-shot setting, we find that adding demonstration examples substantially helps instructed LLMs perform better on most code comprehension and generation tasks; however, the examples would sometimes induce unstable or even worse performance. Furthermore, we find widely-used BM25-based shot selection strategy significantly outperforms the basic random selection or fixed selection only on generation problems. Third, for the fine-tuning setting, we find that fine-tuning could further improve the model performance on downstream code comprehension and generation tasks compared to the zero-shot/one-shot performance. In addition, after being fine-tuned on the same downstream task dataset, instructed LLMs outperform both the small SOTA models and similar-scaled LLMs without instruction tuning. Based on our findings, we further present practical implications on model and usage recommendation, performance and cost trade-offs, and future direction.

翻译：本研究评估了10个开源指令微调大语言模型在四项代表性代码理解与生成任务上的表现。主要发现如下：第一，在零样本设置下，指令微调大语言模型在代码理解与生成任务中表现出极具竞争力的性能，有时甚至优于专门针对下游任务微调的小型最佳模型。同时发现，更大的指令微调大语言模型在代码相关任务中并非总是表现更优。第二，在少样本设置下，添加示例显著提升了指令微调大语言模型在大多数代码理解与生成任务上的性能；但示例有时会导致性能不稳定甚至下降。此外，基于BM25的常用样本选择策略仅在生成问题上显著优于随机选择或固定选择策略。第三，在微调设置下，与零样本/单样本性能相比，微调能进一步提升模型在下游代码理解与生成任务上的表现。同时，在相同下游任务数据集上微调后，指令微调大语言模型不仅超越小型最佳模型，也优于同规模未经过指令微调的大语言模型。基于上述发现，我们进一步提出了关于模型与使用建议、性能与成本权衡以及未来方向的实践启示。