Pre-trained large language models (LLMs) have significantly improved code generation. As these models scale up, there is an increasing need for the output to handle more intricate tasks and to be appropriately specialized to particular domains. Here, we target bioinformatics due to the amount of domain knowledge, algorithms, and data operations this discipline requires. We present BioCoder, a benchmark developed to evaluate LLMs in generating bioinformatics-specific code. BioCoder spans much of the field, covering cross-file dependencies, class declarations, and global variables. It incorporates 1,026 Python functions and 1,243 Java methods extracted from GitHub, along with 253 examples from the Rosalind Project, all pertaining to bioinformatics. Using topic modeling, we show that the overall coverage of the included code is representative of the full spectrum of bioinformatics calculations. BioCoder incorporates a fuzz-testing framework for evaluation. We have applied it to evaluate various models including InCoder, CodeGen, CodeGen2, SantaCoder, StarCoder, StarCoder+, InstructCodeT5+, GPT-3.5, and GPT- 4. Furthermore, we fine-tuned one model (StarCoder), demonstrating that our training dataset can enhance the performance on our testing benchmark (by >15% in terms of Pass@K under certain prompt configurations and always >3%). The results highlight two key aspects of successful models: (1) Successful models accommodate a long prompt (> 2,600 tokens) with full context, including functional dependencies. (2) They contain domain-specific knowledge of bioinformatics, beyond just general coding capability. This is evident from the performance gain of GPT-3.5/4 compared to the smaller models on our benchmark (50% vs. up to 25%). Availability and implementation: Code is available at: https://github.com/gersteinlab/biocoder and https://biocoder-benchmark. github.io/.

翻译：预训练大语言模型（LLMs）显著提升了代码生成能力。随着模型规模不断扩大，输出结果需能处理更复杂的任务，并需针对特定领域进行适当专业化。本文以生物信息学为研究目标，因其涉及大量领域知识、算法及数据操作。我们提出BioCoder——一个专为评估LLMs在生物信息学特定代码生成能力而开发的基准测试。BioCoder覆盖该领域广泛内容，涵盖跨文件依赖、类声明与全局变量。其包含从GitHub提取的1,026个Python函数和1,243个Java方法，以及Rosalind项目中的253个示例，均与生物信息学相关。通过主题建模分析，表明所包含代码的整体覆盖率可代表生物信息学计算的完整范畴。BioCoder采用模糊测试框架进行评估，已应用于包括InCoder、CodeGen、CodeGen2、SantaCoder、StarCoder、StarCoder+、InstructCodeT5+、GPT-3.5和GPT-4在内的多种模型。此外，我们对其中一种模型（StarCoder）进行微调，证明训练数据集可提升测试基准的性能（在特定提示配置下Pass@K提升超15%，且始终大于3%）。结果揭示了成功模型的两个关键特征：（1）成功模型能处理包含完整上下文的长提示（超过2,600个token），包括函数依赖关系；（2）它们具备超越通用编码能力的生物信息学领域特定知识。这一点从GPT-3.5/4相较于较小模型在该基准上的性能增益（50% vs. 最高25%）中得以证实。可用性与实现：代码已开源至https://github.com/gersteinlab/biocoder 及 https://biocoder-benchmark.github.io/。