With the success of large-scale pretraining in NLP, there is an increasing trend of applying it to the domain of life sciences. In particular, pretraining methods based on DNA sequences have garnered growing attention due to their potential to capture generic information about genes. However, existing pretraining methods for DNA sequences largely rely on direct adoptions of BERT pretraining from NLP, lacking a comprehensive understanding and a specifically tailored approach. To address this research gap, we first conducted a series of exploratory experiments and gained several insightful observations: 1) In the fine-tuning phase of downstream tasks, when using K-mer overlapping tokenization instead of K-mer non-overlapping tokenization, both overlapping and non-overlapping pretraining weights show consistent performance improvement.2) During the pre-training process, using K-mer overlapping tokenization quickly produces clear K-mer embeddings and reduces the loss to a very low level, while using K-mer non-overlapping tokenization results in less distinct embeddings and continuously decreases the loss. 3) Using overlapping tokenization causes the self-attention in the intermediate layers of pre-trained models to tend to overly focus on certain tokens, reflecting that these layers are not adequately optimized. In summary, overlapping tokenization can benefit the fine-tuning of downstream tasks but leads to inadequate pretraining with fast convergence. To unleash the pretraining potential, we introduce a novel approach called RandomMask, which gradually increases the task difficulty of BERT-like pretraining by continuously expanding its mask boundary, forcing the model to learn more knowledge. RandomMask is simple but effective, achieving top-tier performance across 26 datasets of 28 datasets spanning 7 downstream tasks.

翻译：随着大规模预训练在自然语言处理领域的成功，其在生命科学领域的应用也日益增多。特别是基于DNA序列的预训练方法，因其能够捕获基因的通用信息而受到广泛关注。然而，现有的DNA序列预训练方法主要直接沿用自然语言处理中的BERT预训练范式，缺乏深入理解与针对性设计。为填补这一研究空白，我们首先开展了一系列探索性实验，获得了若干重要发现：1）在下游任务的微调阶段，当采用K-mer重叠分词而非K-mer非重叠分词时，重叠与非重叠预训练权重均表现出持续的性能提升；2）在预训练过程中，使用K-mer重叠分词能快速生成清晰的K-mer嵌入，并将损失降至极低水平，而使用K-mer非重叠分词则导致嵌入区分度不足且损失持续下降；3）使用重叠分词会导致预训练模型中间层的自注意力机制过度集中于某些词元，表明这些层未得到充分优化。综上，重叠分词有助于下游任务的微调，但会因快速收敛导致预训练不充分。为释放预训练潜力，我们提出名为RandomMask的新方法，通过持续扩大掩码边界逐步增加类BERT预训练的任务难度，迫使模型学习更多知识。RandomMask简单而有效，在涵盖7项下游任务的28个数据集中的26个数据集中达到了顶尖性能。