Protein representation learning methods have shown great potential to yield useful representation for many downstream tasks, especially on protein classification. Moreover, a few recent studies have shown great promise in addressing insufficient labels of proteins with self-supervised learning methods. However, existing protein language models are usually pretrained on protein sequences without considering the important protein structural information. To this end, we propose a novel structure-aware protein self-supervised learning method to effectively capture structural information of proteins. In particular, a well-designed graph neural network (GNN) model is pretrained to preserve the protein structural information with self-supervised tasks from a pairwise residue distance perspective and a dihedral angle perspective, respectively. Furthermore, we propose to leverage the available protein language model pretrained on protein sequences to enhance the self-supervised learning. Specifically, we identify the relation between the sequential information in the protein language model and the structural information in the specially designed GNN model via a novel pseudo bi-level optimization scheme. Experiments on several supervised downstream tasks verify the effectiveness of our proposed method.

翻译：蛋白质表示学习方法在诸多下游任务（尤其是蛋白质分类任务）中展现了生成有用表示的潜力。此外，近期一些研究在利用自监督学习方法解决蛋白质标记不足问题方面显示出巨大前景。然而，现有蛋白质语言模型通常仅基于蛋白质序列进行预训练，而未考虑重要的蛋白质结构信息。为此，我们提出一种新颖的结构感知蛋白质自监督学习方法，以有效捕获蛋白质的结构信息。具体而言，我们预训练了一个精心设计的图神经网络（GNN）模型，通过分别从成对残基距离和二面角角度两个视角的自监督任务，保留蛋白质结构信息。此外，我们提出利用已在蛋白质序列上预训练的蛋白质语言模型来增强自监督学习。具体来说，我们通过一种新颖的伪双层优化方案，识别出蛋白质语言模型中的序列信息与特殊设计的GNN模型中的结构信息之间的关系。在多个监督下游任务上的实验验证了我们所提出方法的有效性。