DNA language models have emerged as powerful tools for decoding the complex language of DNA sequences. However, the performance of these models is heavily affected by their tokenization strategy, i.e., a method used to parse DNA sequences into a shorter sequence of chunks. In this work, we propose DNACHUNKER, which integrates a learnable dynamic DNA tokenization mechanism and is trained as a masked language model. Adopting the dynamic chunking procedure proposed by H-Net, our model learns to segment sequences into variable-length chunks. This dynamic chunking offers two key advantages: it's resilient to shifts and mutations in the DNA, and it allocates more detail to important functional areas. We demonstrate the performance of DNACHUNKER by training it on the human reference genome (HG38) and testing it on the Nucleotide Transformer and Genomic benchmarks. Further ablative experiments reveal that DNACHUNKER learns tokenization that grasps biological grammar and uses smaller chunks to preserve detail in important functional elements such as promoters and exons, while using larger chunks for repetitive, redundant regions.

翻译：DNA语言模型已成为解码DNA序列复杂语言的有力工具。然而，这些模型的性能在很大程度上受其分词策略的影响，即用于将DNA序列解析为更短片段序列的方法。本文提出DNACHUNKER，它集成了可学习的动态DNA分词机制，并以掩码语言模型进行训练。通过采用H-Net提出的动态分块流程，我们的模型学习将序列分割为可变长度的片段。这种动态分块具有两个关键优势：对DNA中的偏移和突变具有鲁棒性，并且能为重要的功能区域分配更多细节。我们通过在人类参考基因组（HG38）上训练DNACHUNKER，并在Nucleotide Transformer和Genomic基准测试上进行评估，证明了其性能。进一步的消融实验表明，DNACHUNKER学习到的分词方法能够把握生物语法，并使用较小的片段来保留重要功能元件（如启动子和外显子）的细节，同时对重复、冗余区域使用较大的片段。