In recent years, there has been a surge in the development of 3D structure-based pre-trained protein models, representing a significant advancement over pre-trained protein language models in various downstream tasks. However, most existing structure-based pre-trained models primarily focus on the residue level, i.e., alpha carbon atoms, while ignoring other atoms like side chain atoms. We argue that modeling proteins at both residue and atom levels is important since the side chain atoms can also be crucial for numerous downstream tasks, for example, molecular docking. Nevertheless, we find that naively combining residue and atom information during pre-training typically fails. We identify a key reason is the information leakage caused by the inclusion of atom structure in the input, which renders residue-level pre-training tasks trivial and results in insufficiently expressive residue representations. To address this issue, we introduce a span mask pre-training strategy on 3D protein chains to learn meaningful representations of both residues and atoms. This leads to a simple yet effective approach to learning protein representation suitable for diverse downstream tasks. Extensive experimental results on binding site prediction and function prediction tasks demonstrate our proposed pre-training approach significantly outperforms other methods. Our code will be made public.

翻译：近年来，基于三维结构的蛋白质预训练模型发展迅速，在各类下游任务中显著超越了预训练蛋白质语言模型。然而，现有结构预训练模型主要关注残基层级（即α碳原子），忽略了侧链原子等其他原子。我们论证，由于侧链原子在分子对接等众多下游任务中具有关键作用，在残基与原子双层级建模蛋白质具有重要意义。但研究发现，简单将残基与原子信息进行联合预训练通常失效。究其原因，输入中包含原子结构会导致信息泄露，使残基层级预训练任务失去意义，进而导致残基表征表达能力不足。为解决该问题，我们提出在三维蛋白质链上采用跨度掩码预训练策略，以学习残基与原子的有效表征。由此形成一套简单高效的方法，能够学习适用于多种下游任务的蛋白质表征。在结合位点预测和功能预测任务上的大量实验表明，我们提出的预训练方法显著优于其他方法。相关代码将开源。