Proteins, essential to biological systems, perform functions intricately linked to their three-dimensional structures. Understanding the relationship between protein structures and their amino acid sequences remains a core challenge in protein modeling. While traditional protein foundation models benefit from pre-training on vast unlabeled datasets, they often struggle to capture critical co-evolutionary information, which evolutionary-based methods excel at. In this study, we introduce a novel pre-training strategy for protein foundation models that emphasizes the interactions among amino acid residues to enhance the extraction of both short-range and long-range co-evolutionary features from sequence data. Trained on a large-scale protein sequence dataset, our model demonstrates superior generalization ability, outperforming established baselines of similar size, including the ESM model, across diverse downstream tasks. Experimental results confirm the model's effectiveness in integrating co-evolutionary information, marking a significant step forward in protein sequence-based modeling.

翻译：蛋白质作为生物系统的核心要素，其功能与三维结构密切相关。理解蛋白质结构与其氨基酸序列之间的关系，始终是蛋白质建模领域的核心挑战。传统的蛋白质基础模型虽受益于海量无标注数据的预训练，却常难以捕捉关键的协同进化信息，而这正是基于进化方法所擅长的。本研究提出一种新颖的蛋白质基础模型预训练策略，通过强调氨基酸残基间的相互作用，从序列数据中增强提取短程与长程协同进化特征的能力。基于大规模蛋白质序列数据集训练后，本模型展现出卓越的泛化性能，在多种下游任务中均超越了包括ESM模型在内的同类规模基准模型。实验结果证实了该模型在整合协同进化信息方面的有效性，标志着基于蛋白质序列的建模研究取得了重要进展。