Publicly available source-code libraries are continuously growing and changing. This makes it impossible for models of code to keep current with all available APIs by simply training these models on existing code repositories. Thus, existing models inherently cannot generalize to using unseen functions and libraries, because these would never appear in the training data. In contrast, when human programmers use functions and libraries for the first time, they frequently refer to textual resources such as code manuals and documentation, to explore and understand the available functionality. Inspired by this observation, we introduce DocPrompting: a natural-language-to-code generation approach that explicitly leverages documentation by (1) retrieving the relevant documentation pieces given an NL intent, and (2) generating code based on the NL intent and the retrieved documentation. DocPrompting is general: it can be applied to any programming language and is agnostic to the underlying neural model. We demonstrate that DocPrompting consistently improves NL-to-code models: DocPrompting improves strong base models such as CodeT5 by 2.85% in pass@1 (52% relative gain) and 4.39% in pass@10 (30% relative gain) in execution-based evaluation on the popular Python CoNaLa benchmark; on a new Bash dataset tldr, DocPrompting improves CodeT5 and GPT-Neo1.3B by up to absolute 6.9% exact match.

翻译：公开可用的源代码库持续增长并不断变化，这使得仅通过训练现有代码仓库的模型无法跟上所有可用API的更新步伐。因此，现有模型本质上无法泛化到未见过的函数和库，因为这些内容永远不会出现在训练数据中。相比之下，当人类程序员首次使用函数和库时，他们经常参考代码手册和文档等文本资源来探索和理解可用功能。受此观察启发，我们提出DocPrompting：一种自然语言到代码的生成方法，通过以下步骤显式利用文档：（1）根据自然语言意图检索相关文档片段，（2）基于自然语言意图和检索到的文档生成代码。DocPrompting具有通用性：可应用于任何编程语言，且与底层神经模型无关。我们证明DocPrompting能持续提升自然语言到代码模型：在基于执行的评估中，DocPrompting在流行的Python CoNaLa基准上将CodeT5等强基模型在pass@1指标上提升2.85%（相对增益52%），在pass@10指标上提升4.39%（相对增益30%）；在新型Bash数据集tldr上，DocPrompting使CodeT5和GPT-Neo1.3B的精确匹配率提升高达6.9%。