Large language models (LLMs) such as ChatGPT have shown remarkable capabilities in code generation. Despite the great achievement, they rely on enormous training data to acquire a broad spectrum of open-domain knowledge. Besides, their evaluation revolves around open-domain benchmarks like HumanEval, which primarily consist of programming contests. Therefore, it is hard to fully characterize the intricacies and challenges associated with particular domains (e.g., web, game, and math). In this paper, we conduct an in-depth study of the LLMs in domain-specific code generation. Our results demonstrate that LLMs exhibit sub-optimal performance in generating domain-specific code, due to their limited proficiency in utilizing domain-specific libraries. We further observe that incorporating API knowledge as prompts can empower LLMs to generate more professional code. Based on these findings, we further investigate how to efficiently incorporate API knowledge into the code generation process. We experiment with three strategies for incorporating domain knowledge, namely, external knowledge inquirer, chain-of-thought prompting, and chain-of-thought fine-tuning. We refer to these strategies as a new code generation approach called DomCoder. Experimental results show that all strategies of DomCoder lead to improvement in the effectiveness of domain-specific code generation under certain settings. The results also show that there is still ample room for further improvement, based on which we suggest possible future works.

翻译：大语言模型（如ChatGPT）在代码生成方面展现出卓越能力。尽管取得显著成就，但这些模型依赖海量训练数据获取广泛的开放领域知识。此外，其评估主要基于HumanEval等以编程竞赛为主的开放领域基准测试，因此难以充分刻画特定领域（如Web、游戏和数学）的复杂性与挑战性。本文对大语言模型在领域特定代码生成中的表现进行了深入研究。结果表明，由于大语言模型在利用领域特定库方面存在局限性，其在生成领域特定代码时表现欠佳。我们进一步观察到，将API知识作为提示融入生成过程可增强模型生成更专业代码的能力。基于这一发现，我们进一步探究如何高效地将API知识融入代码生成过程，实验采用三种领域知识融入策略：外部知识查询器、思维链提示与思维链微调。我们将这些策略统称为新型代码生成方法DomCoder。实验结果表明，在特定设置下，DomCoder的所有策略均能提升领域特定代码生成的有效性。这些结果也表明当前仍存在充足的性能提升空间，并据此提出未来可能的研究方向。