Automated computational analysis of the vast chemical space is critical for numerous fields of research such as drug discovery and material science. Representation learning techniques have recently been employed with the primary objective of generating compact and informative numerical expressions of complex data. One approach to efficiently learn molecular representations is processing string-based notations of chemicals via natural language processing (NLP) algorithms. Majority of the methods proposed so far utilize SMILES notations for this purpose; however, SMILES is associated with numerous problems related to validity and robustness, which may prevent the model from effectively uncovering the knowledge hidden in the data. In this study, we propose SELFormer, a transformer architecture-based chemical language model that utilizes a 100% valid, compact and expressive notation, SELFIES, as input, in order to learn flexible and high-quality molecular representations. SELFormer is pre-trained on two million drug-like compounds and fine-tuned for diverse molecular property prediction tasks. Our performance evaluation has revealed that, SELFormer outperforms all competing methods, including graph learning-based approaches and SMILES-based chemical language models, on predicting aqueous solubility of molecules and adverse drug reactions. We also visualized molecular representations learned by SELFormer via dimensionality reduction, which indicated that even the pre-trained model can discriminate molecules with differing structural properties. We shared SELFormer as a programmatic tool, together with its datasets and pre-trained models. Overall, our research demonstrates the benefit of using the SELFIES notations in the context of chemical language modeling and opens up new possibilities for the design and discovery of novel drug candidates with desired features.

翻译：对广阔化学空间进行自动化计算分析对于药物发现和材料科学等众多研究领域至关重要。近年来，表征学习技术被广泛应用于生成复杂数据的紧凑且信息丰富的数值表达。通过自然语言处理（NLP）算法处理基于字符串的化学符号是高效学习分子表示的方法之一。目前提出的多数方法均采用SMILES符号进行此类任务；然而，SMILES存在与有效性和鲁棒性相关的诸多问题，可能阻碍模型有效挖掘数据中隐藏的知识。在本研究中，我们提出SELFormer——一种基于Transformer架构的化学语言模型，其采用100%有效、紧凑且表达能力强的SELFIES符号作为输入，以学习灵活且高质量的分子表示。SELFormer在200万个类药化合物上完成预训练，并针对多种分子性质预测任务进行微调。性能评估表明，SELFormer在预测分子水溶性及药物不良反应方面优于所有竞争方法，包括基于图学习的方法和基于SMILES的化学语言模型。我们还通过降维可视化研究了SELFormer习得的分子表示，结果显示即使预训练模型也能区分具有不同结构特性的分子。我们将SELFormer作为程序化工具开放共享，并附带其数据集和预训练模型。总体而言，本研究表明在化学语言建模中采用SELFIES符号的显著优势，并为设计和发现具有所需特性的新型候选药物开辟了新途径。