The parallels between protein sequences and natural language in their sequential structures have inspired the application of large language models (LLMs) to protein understanding. Despite the success of LLMs in NLP, their effectiveness in comprehending protein sequences remains an open question, largely due to the absence of datasets linking protein sequences to descriptive text. Researchers have then attempted to adapt LLMs for protein understanding by integrating a protein sequence encoder with a pre-trained LLM. However, this adaptation raises a fundamental question: "Can LLMs, originally designed for NLP, effectively comprehend protein sequences as a form of language?" Current datasets fall short in addressing this question due to the lack of a direct correlation between protein sequences and corresponding text descriptions, limiting the ability to train and evaluate LLMs for protein understanding effectively. To bridge this gap, we introduce ProteinLMDataset, a dataset specifically designed for further self-supervised pretraining and supervised fine-tuning (SFT) of LLMs to enhance their capability for protein sequence comprehension. Specifically, ProteinLMDataset includes 17.46 billion tokens for pretraining and 893,000 instructions for SFT. Additionally, we present ProteinLMBench, the first benchmark dataset consisting of 944 manually verified multiple-choice questions for assessing the protein understanding capabilities of LLMs. ProteinLMBench incorporates protein-related details and sequences in multiple languages, establishing a new standard for evaluating LLMs' abilities in protein comprehension. The large language model InternLM2-7B, pretrained and fine-tuned on the ProteinLMDataset, outperforms GPT-4 on ProteinLMBench, achieving the highest accuracy score.

翻译：蛋白质序列与自然语言在序列结构上的相似性启发研究者将大语言模型应用于蛋白质理解领域。尽管LLMs在自然语言处理中取得显著成功，但其在蛋白质序列理解方面的有效性仍存疑问，这主要源于缺乏关联蛋白质序列与描述性文本的数据集。为此，研究者尝试通过整合蛋白质序列编码器与预训练LLM来适配蛋白质理解任务。然而，这种适配引发了一个根本性问题：“为自然语言处理设计的LLMs能否将蛋白质序列作为一种语言进行有效理解？”现有数据集因缺乏蛋白质序列与对应文本描述间的直接关联，限制了有效训练与评估蛋白质理解LLMs的能力，难以回答该问题。为填补这一空白，我们提出了ProteinLMDataset——一个专门用于LLMs进一步自监督预训练与监督微调的数据集，旨在增强模型对蛋白质序列的理解能力。具体而言，ProteinLMDataset包含174.6亿个预训练标记和89.3万条SFT指令。此外，我们发布了ProteinLMBench——首个包含944道人工验证多选题的基准数据集，用于评估LLMs的蛋白质理解能力。该基准融合了多语言蛋白质相关描述与序列，为评估LLMs的蛋白质理解能力设立了新标准。基于ProteinLMDataset进行预训练与微调的大语言模型InternLM2-7B，在ProteinLMBench上超越了GPT-4，取得了最高准确率得分。