The use of deep learning models in computational biology has increased massively in recent years, and is expected to do so further with the current advances in fields like Natural Language Processing. These models, although able to draw complex relations between input and target, are also largely inclined to learn noisy deviations from the pool of data used during their development. In order to assess their performance on unseen data (their capacity to generalize), it is common to randomly split the available data in development (train/validation) and test sets. This procedure, although standard, has lately been shown to produce dubious assessments of generalization due to the existing similarity between samples in the databases used. In this work, we present SpanSeq, a database partition method for machine learning that can scale to most biological sequences (genes, proteins and genomes) in order to avoid data leakage between sets. We also explore the effect of not restraining similarity between sets by reproducing the development of the state-of-the-art model DeepLoc, not only confirming the consequences of randomly splitting databases on the model assessment, but expanding those repercussions to the model development. SpanSeq is available for downloading and installing at https://github.com/genomicepidemiology/SpanSeq.

翻译：近年来，深度学习模型在计算生物学中的应用大幅增加，且预计随着自然语言处理等领域的当前进展将进一步增长。这类模型虽能建立输入与目标之间的复杂关联，但也极易从开发所用的数据池中学习噪声偏差。为评估其在未见数据上的表现（即泛化能力），通常会将可用数据随机划分为开发集（训练/验证集）和测试集。尽管此流程为标准做法，但近期研究表明，由于所用数据库中样本间存在相似性，该方法可能产生不可靠的泛化评估。本文提出SpanSeq——一种适用于机器学习的数据集划分方法，可扩展至大多数生物序列（基因、蛋白质和基因组），从而避免集合间的数据泄露。我们通过复现最先进模型DeepLoc的开发过程，探究了不限制集合间相似性的影响，不仅证实了随机划分数据库对模型评估的后果，还揭示了这些影响延伸至模型开发阶段。SpanSeq可在https://github.com/genomicepidemiology/SpanSeq下载安装。